xfs 在各种 RAID 下的性能测试

1. 磁盘基准测试

1.1 4k 随机写

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
fio \
  --name=4k_randwrite \
  --rw=randwrite \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/data1/fio-4k-randwrite.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][r=0KiB/s,w=252MiB/s][r=0,w=64.5k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=31269
  write: IOPS=65.1k, BW=254MiB/s (267MB/s)(29.8GiB/120004msec)
    slat (usec): min=2, max=300387, avg=98.81, stdev=642.76
    clat (nsec): min=514, max=326192k, avg=3833374.99, stdev=5344771.80
     lat (usec): min=15, max=326239, avg=3932.32, stdev=5437.71
    clat percentiles (usec):
     |  1.00th=[  217],  5.00th=[  693], 10.00th=[  857], 20.00th=[ 1139],
     | 30.00th=[ 1418], 40.00th=[ 1811], 50.00th=[ 2376], 60.00th=[ 3130],
     | 70.00th=[ 4015], 80.00th=[ 5145], 90.00th=[ 7308], 95.00th=[11338],
     | 99.00th=[25822], 99.50th=[33817], 99.90th=[56886], 99.95th=[66323],
     | 99.99th=[87557]
   bw (  KiB/s): min= 7398, max=81995, per=12.51%, avg=32563.03, stdev=6393.43, samples=1920
   iops        : min= 1849, max=20498, avg=8140.60, stdev=1598.37, samples=1920
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.05%, 50=0.19%
  lat (usec)   : 100=0.22%, 250=0.72%, 500=1.61%, 750=3.68%, 1000=8.40%
  lat (msec)   : 2=28.96%, 4=26.09%, 10=24.03%, 20=4.27%, 50=1.63%
  lat (msec)   : 100=0.16%, 250=0.01%, 500=0.01%
  cpu          : usr=2.08%, sys=34.78%, ctx=7037215, majf=0, minf=10314
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,7809386,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=254MiB/s (267MB/s), 254MiB/s-254MiB/s (267MB/s-267MB/s), io=29.8GiB (31.0GB), run=120004-120004msec

Disk stats (read/write):
  nvme0n1: ios=0/7820641, merge=0/293550, ticks=0/10477516, in_queue=11478540, util=53.82%

1.2 4k 随机读

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4k_randread \
  --rw=randread \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --filename=/data1/fio-4k-randread.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [r(8)][100.0%][r=1140MiB/s,w=0KiB/s][r=292k,w=0 IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=58294
   read: IOPS=294k, BW=1147MiB/s (1202MB/s)(134GiB/120006msec)
    slat (nsec): min=1697, max=23878k, avg=5790.82, stdev=16187.41
    clat (usec): min=2, max=26723, avg=864.94, stdev=780.57
     lat (usec): min=43, max=26728, avg=870.84, stdev=780.71
    clat percentiles (usec):
     |  1.00th=[  105],  5.00th=[  135], 10.00th=[  182], 20.00th=[  273],
     | 30.00th=[  375], 40.00th=[  494], 50.00th=[  635], 60.00th=[  799],
     | 70.00th=[ 1012], 80.00th=[ 1319], 90.00th=[ 1844], 95.00th=[ 2376],
     | 99.00th=[ 3687], 99.50th=[ 4293], 99.90th=[ 5866], 99.95th=[ 6652],
     | 99.99th=[ 9241]
   bw (  KiB/s): min=130360, max=164784, per=12.52%, avg=146966.37, stdev=2595.35, samples=1918
   iops        : min=32590, max=41196, avg=36741.37, stdev=648.83, samples=1918
  lat (usec)   : 4=0.01%, 20=0.01%, 50=0.01%, 100=0.28%, 250=17.26%
  lat (usec)   : 500=22.94%, 750=16.82%, 1000=12.10%
  lat (msec)   : 2=22.44%, 4=7.48%, 10=0.68%, 20=0.01%, 50=0.01%
  cpu          : usr=10.07%, sys=29.47%, ctx=19757756, majf=0, minf=8027
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=35228529,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1147MiB/s (1202MB/s), 1147MiB/s-1147MiB/s (1202MB/s-1202MB/s), io=134GiB (144GB), run=120006-120006msec

Disk stats (read/write):
  nvme0n1: ios=35183593/4, merge=0/1, ticks=29535284/0, in_queue=33076828, util=100.00%

1.3 4k 随机读写

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
fio \
  --name=4k_randrw \
  --rw=randrw \
  --rwmixread=70 \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=1 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/data1/fio-4k-randrw.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=178MiB/s,w=76.4MiB/s][r=45.7k,w=19.6k IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=39459
   read: IOPS=45.0k, BW=180MiB/s (188MB/s)(21.0GiB/120001msec)
    slat (usec): min=2, max=11092, avg= 6.59, stdev=23.78
    clat (nsec): min=995, max=23512k, avg=155824.83, stdev=117179.25
     lat (usec): min=15, max=23538, avg=162.52, stdev=119.69
    clat percentiles (usec):
     |  1.00th=[   87],  5.00th=[   92], 10.00th=[   98], 20.00th=[  103],
     | 30.00th=[  109], 40.00th=[  118], 50.00th=[  135], 60.00th=[  151],
     | 70.00th=[  165], 80.00th=[  184], 90.00th=[  231], 95.00th=[  273],
     | 99.00th=[  424], 99.50th=[  742], 99.90th=[ 1663], 99.95th=[ 2008],
     | 99.99th=[ 3490]
   bw (  KiB/s): min=15960, max=24321, per=12.52%, avg=23011.22, stdev=652.38, samples=1916
   iops        : min= 3990, max= 6080, avg=5752.59, stdev=163.06, samples=1916
  write: IOPS=19.7k, BW=76.0MiB/s (80.7MB/s)(9236MiB/120001msec)
    slat (usec): min=2, max=7925, avg= 8.38, stdev=23.90
    clat (nsec): min=516, max=11228k, avg=13944.32, stdev=29810.73
     lat (usec): min=11, max=11286, avg=22.44, stdev=38.63
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   11], 10.00th=[   11], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   13], 50.00th=[   13], 60.00th=[   14],
     | 70.00th=[   14], 80.00th=[   15], 90.00th=[   16], 95.00th=[   19],
     | 99.00th=[   34], 99.50th=[   48], 99.90th=[   96], 99.95th=[  131],
     | 99.99th=[  955]
   bw (  KiB/s): min= 7080, max=11076, per=12.52%, avg=9865.67, stdev=419.59, samples=1916
   iops        : min= 1770, max= 2769, avg=2466.18, stdev=104.96, samples=1916
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.19%, 20=28.70%, 50=0.96%
  lat (usec)   : 100=10.01%, 250=55.07%, 500=4.51%, 750=0.18%, 1000=0.12%
  lat (msec)   : 2=0.20%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=2.69%, sys=8.01%, ctx=8499979, majf=0, minf=1178
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=5514868,2364483,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=180MiB/s (188MB/s), 180MiB/s-180MiB/s (188MB/s-188MB/s), io=21.0GiB (22.6GB), run=120001-120001msec
  WRITE: bw=76.0MiB/s (80.7MB/s), 76.0MiB/s-76.0MiB/s (80.7MB/s-80.7MB/s), io=9236MiB (9685MB), run=120001-120001msec

Disk stats (read/write):
  nvme0n1: ios=5513305/2363838, merge=0/23, ticks=834508/27104, in_queue=739972, util=100.00%

1.4 4M 顺序写

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4m_seq_write \
  --rw=write \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/data1

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=921MiB/s][r=0,w=230 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=60739
  write: IOPS=237, BW=950MiB/s (996MB/s)(55.7GiB/60038msec)
    slat (usec): min=342, max=15127, avg=747.13, stdev=315.68
    clat (msec): min=3, max=228, avg=32.93, stdev=13.08
     lat (msec): min=4, max=229, avg=33.68, stdev=13.07
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   17], 10.00th=[   18], 20.00th=[   21],
     | 30.00th=[   25], 40.00th=[   29], 50.00th=[   32], 60.00th=[   34],
     | 70.00th=[   39], 80.00th=[   44], 90.00th=[   51], 95.00th=[   57],
     | 99.00th=[   70], 99.50th=[   74], 99.90th=[   80], 99.95th=[   85],
     | 99.99th=[  228]
   bw (  KiB/s): min=925696, max=1007616, per=100.00%, avg=972891.75, stdev=19806.08, samples=120
   iops        : min=  226, max=  246, avg=237.47, stdev= 4.82, samples=120
  lat (msec)   : 4=0.01%, 10=0.03%, 20=16.83%, 50=72.57%, 100=10.54%
  lat (msec)   : 250=0.03%
  cpu          : usr=8.25%, sys=9.91%, ctx=13396, majf=0, minf=16435
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,14256,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=950MiB/s (996MB/s), 950MiB/s-950MiB/s (996MB/s-996MB/s), io=55.7GiB (59.8GB), run=60038-60038msec

Disk stats (read/write):
  nvme0n1: ios=0/454702, merge=0/834, ticks=0/12245780, in_queue=12288772, util=100.00%

4M 顺序写多进程和单进程数据差不多。

1.5 4M 顺序读

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4m_seq_read \
  --rw=read \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/data1

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=2919MiB/s,w=0KiB/s][r=729,w=0 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=39149
   read: IOPS=722, BW=2890MiB/s (3031MB/s)(169GiB/60008msec)
    slat (usec): min=170, max=6405, avg=388.71, stdev=172.35
    clat (usec): min=2039, max=35296, avg=10676.32, stdev=1174.56
     lat (usec): min=3152, max=40049, avg=11065.58, stdev=1163.86
    clat percentiles (usec):
     |  1.00th=[ 7635],  5.00th=[ 8717], 10.00th=[ 9110], 20.00th=[ 9765],
     | 30.00th=[10159], 40.00th=[10552], 50.00th=[10814], 60.00th=[11076],
     | 70.00th=[11338], 80.00th=[11600], 90.00th=[11994], 95.00th=[12256],
     | 99.00th=[13042], 99.50th=[13435], 99.90th=[15533], 99.95th=[16188],
     | 99.99th=[28443]
   bw (  MiB/s): min= 2720, max= 3160, per=100.00%, avg=2890.76, stdev=149.46, samples=120
   iops        : min=  680, max=  790, avg=722.66, stdev=37.37, samples=120
  lat (msec)   : 4=0.03%, 10=26.26%, 20=73.69%, 50=0.02%
  cpu          : usr=0.49%, sys=28.86%, ctx=41669, majf=0, minf=24632
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=43363,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=2890MiB/s (3031MB/s), 2890MiB/s-2890MiB/s (3031MB/s-3031MB/s), io=169GiB (182GB), run=60008-60008msec

Disk stats (read/write):
  nvme0n1: ios=1382412/5, merge=0/1, ticks=7162440/0, in_queue=7266616, util=100.00%

4M 顺序读多进程和单进程数据差不多。

2. RAID 0

2.1 创建 RAID 0

查看磁盘

1
lsblk -d -o NAME,SIZE,TYPE | grep nvme

1
2
3
4
5
6
nvme0n1 745.2G disk
nvme1n1 745.2G disk
nvme2n1 745.2G disk
nvme3n1 745.2G disk
nvme4n1 745.2G disk
nvme5n1 745.2G disk

创建 RAID

1
mdadm --create --verbose /dev/md0 --level=0 --raid-devices=5 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1

这里的 level=0 表示 RAID 0，raid-devices=5 表示使用 5 块盘。

查看 RAID 状态

1
mdadm --detail /dev/md0

等盘 State : clean 再操作。

格式化 RAID

1
mkfs.xfs -f /dev/md0

挂载 RAID

1
2
mkdir -p /mnt/mdadm
mount /dev/md0 /mnt/mdadm

查看磁盘大小

1
df -h /mnt/mdadm/

1
2
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        3.7T  3.8G  3.7T   1% /mnt/mdadm

2.2 4k 随机写

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
fio \
  --name=4k_randwrite \
  --rw=randwrite \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randwrite.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][r=0KiB/s,w=471MiB/s][r=0,w=121k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=21458
  write: IOPS=87.9k, BW=343MiB/s (360MB/s)(40.2GiB/120001msec)
    slat (usec): min=3, max=480801, avg=87.68, stdev=558.72
    clat (nsec): min=1023, max=495513k, avg=2824145.32, stdev=4122691.21
     lat (usec): min=15, max=495562, avg=2911.98, stdev=4197.27
    clat percentiles (usec):
     |  1.00th=[  537],  5.00th=[  799], 10.00th=[  914], 20.00th=[ 1090],
     | 30.00th=[ 1254], 40.00th=[ 1434], 50.00th=[ 1663], 60.00th=[ 1975],
     | 70.00th=[ 2507], 80.00th=[ 3556], 90.00th=[ 5997], 95.00th=[ 8979],
     | 99.00th=[17171], 99.50th=[21103], 99.90th=[30802], 99.95th=[34866],
     | 99.99th=[48497]
   bw (  KiB/s): min= 2076, max=104449, per=12.52%, avg=43999.32, stdev=19351.22, samples=1920
   iops        : min=  519, max=26112, avg=10999.54, stdev=4837.80, samples=1920
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.04%, 500=0.74%, 750=2.84%, 1000=11.08%
  lat (msec)   : 2=45.76%, 4=22.23%, 10=13.25%, 20=3.45%, 50=0.60%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=2.51%, sys=47.82%, ctx=8084183, majf=0, minf=2705
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,10544663,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=343MiB/s (360MB/s), 343MiB/s-343MiB/s (360MB/s-360MB/s), io=40.2GiB (43.2GB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=1/10532623, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/2110560, aggrmerge=0/0, aggrticks=0/17402, aggrin_queue=472, aggrutil=0.41%
  nvme0n1: ios=0/2111164, merge=0/1, ticks=0/18048, in_queue=700, util=0.41%
  nvme3n1: ios=0/2109897, merge=0/0, ticks=0/17404, in_queue=448, util=0.31%
  nvme2n1: ios=0/2108603, merge=0/0, ticks=0/17316, in_queue=412, util=0.30%
  nvme1n1: ios=1/2114000, merge=0/0, ticks=0/17700, in_queue=380, util=0.29%
  nvme4n1: ios=0/2109140, merge=0/0, ticks=0/16544, in_queue=424, util=0.24%

2.3 4k 随机读

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4k_randread \
  --rw=randread \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --filename=/mnt/mdadm/fio-4k-randread.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [r(8)][100.0%][r=3783MiB/s,w=0KiB/s][r=969k,w=0 IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=49074
   read: IOPS=1009k, BW=3940MiB/s (4132MB/s)(462GiB/120002msec)
    slat (nsec): min=1916, max=23433k, avg=6686.82, stdev=13633.31
    clat (usec): min=34, max=23910, avg=246.46, stdev=104.87
     lat (usec): min=38, max=23922, avg=253.20, stdev=106.05
    clat percentiles (usec):
     |  1.00th=[  153],  5.00th=[  172], 10.00th=[  182], 20.00th=[  194],
     | 30.00th=[  206], 40.00th=[  217], 50.00th=[  229], 60.00th=[  245],
     | 70.00th=[  265], 80.00th=[  289], 90.00th=[  334], 95.00th=[  375],
     | 99.00th=[  474], 99.50th=[  519], 99.90th=[  652], 99.95th=[  742],
     | 99.99th=[ 1876]
   bw (  KiB/s): min=323448, max=598204, per=12.51%, avg=504829.01, stdev=36370.66, samples=1920
   iops        : min=80862, max=149551, avg=126207.05, stdev=9092.64, samples=1920
  lat (usec)   : 50=0.01%, 100=0.01%, 250=62.79%, 500=36.53%, 750=0.62%
  lat (usec)   : 1000=0.03%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=16.66%, sys=82.78%, ctx=96372, majf=0, minf=5346
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=121052850,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=3940MiB/s (4132MB/s), 3940MiB/s-3940MiB/s (4132MB/s-4132MB/s), io=462GiB (496GB), run=120002-120002msec

Disk stats (read/write):
    md0: ios=120867592/15, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=24210570/2, aggrmerge=0/0, aggrticks=3085131/0, aggrin_queue=3179908, aggrutil=100.00%
  nvme0n1: ios=24210257/8, merge=0/2, ticks=3083560/0, in_queue=3146512, util=100.00%
  nvme3n1: ios=24209120/0, merge=0/0, ticks=3085116/0, in_queue=3163832, util=100.00%
  nvme2n1: ios=24210502/0, merge=0/0, ticks=3088560/0, in_queue=3243228, util=100.00%
  nvme1n1: ios=24211505/2, merge=0/0, ticks=3076108/0, in_queue=3173424, util=100.00%
  nvme4n1: ios=24211466/3, merge=0/0, ticks=3092312/0, in_queue=3172548, util=100.00%

2.4 4k 随机读写

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
fio \
  --name=4k_randrw \
  --rw=randrw \
  --rwmixread=70 \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=1 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randrw.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=246MiB/s,w=105MiB/s][r=63.1k,w=26.0k IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=42379
   read: IOPS=62.6k, BW=245MiB/s (257MB/s)(28.7GiB/120001msec)
    slat (usec): min=2, max=15986, avg= 8.75, stdev=43.09
    clat (nsec): min=725, max=20233k, avg=100358.72, stdev=59825.88
     lat (usec): min=14, max=23745, avg=109.26, stdev=75.95
    clat percentiles (usec):
     |  1.00th=[   86],  5.00th=[   87], 10.00th=[   88], 20.00th=[   89],
     | 30.00th=[   90], 40.00th=[   92], 50.00th=[   94], 60.00th=[   99],
     | 70.00th=[  105], 80.00th=[  106], 90.00th=[  113], 95.00th=[  127],
     | 99.00th=[  172], 99.50th=[  188], 99.90th=[  302], 99.95th=[  519],
     | 99.99th=[ 2966]
   bw (  KiB/s): min=10072, max=33010, per=12.51%, avg=31349.65, stdev=1699.73, samples=1912
   iops        : min= 2518, max= 8252, avg=7837.21, stdev=424.93, samples=1912
  write: IOPS=26.9k, BW=105MiB/s (110MB/s)(12.3GiB/120001msec)
    slat (usec): min=3, max=20195, avg=13.70, stdev=51.79
    clat (nsec): min=572, max=307733k, avg=22953.37, stdev=287025.71
     lat (usec): min=13, max=307770, avg=36.82, stdev=291.95
    clat percentiles (usec):
     |  1.00th=[   13],  5.00th=[   16], 10.00th=[   16], 20.00th=[   18],
     | 30.00th=[   18], 40.00th=[   19], 50.00th=[   20], 60.00th=[   20],
     | 70.00th=[   21], 80.00th=[   23], 90.00th=[   25], 95.00th=[   30],
     | 99.00th=[   56], 99.50th=[   75], 99.90th=[  289], 99.95th=[  930],
     | 99.99th=[ 4359]
   bw (  KiB/s): min= 4224, max=15014, per=12.51%, avg=13438.51, stdev=799.40, samples=1912
   iops        : min= 1056, max= 3753, avg=3359.39, stdev=199.83, samples=1912
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.04%, 4=0.01%, 10=0.01%, 20=18.55%, 50=11.01%
  lat (usec)   : 100=42.95%, 250=27.27%, 500=0.10%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=4.46%, sys=15.06%, ctx=11607725, majf=0, minf=2890
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=7517356,3222298,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=245MiB/s (257MB/s), 245MiB/s-245MiB/s (257MB/s-257MB/s), io=28.7GiB (30.8GB), run=120001-120001msec
  WRITE: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=12.3GiB (13.2GB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=7510767/3219671, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1503471/644462, aggrmerge=0/0, aggrticks=145287/6216, aggrin_queue=69136, aggrutil=41.33%
  nvme0n1: ios=1502724/643760, merge=0/3, ticks=144384/6196, in_queue=67528, util=40.15%
  nvme3n1: ios=1501741/644320, merge=0/1, ticks=144768/6540, in_queue=69136, util=41.33%
  nvme2n1: ios=1506705/644691, merge=0/0, ticks=146568/6288, in_queue=70480, util=41.25%
  nvme1n1: ios=1503224/645075, merge=0/0, ticks=145016/6036, in_queue=69064, util=41.23%
  nvme4n1: ios=1502962/644466, merge=0/0, ticks=145700/6020, in_queue=69476, util=40.81%

2.5 4M 顺序写

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4m_seq_write \
  --rw=write \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=4844MiB/s][r=0,w=1211 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=27148
  write: IOPS=1203, BW=4815MiB/s (5049MB/s)(282GiB/60001msec)
    slat (usec): min=287, max=7225, avg=681.36, stdev=196.26
    clat (usec): min=411, max=33510, avg=5961.80, stdev=1525.37
     lat (usec): min=921, max=37964, avg=6643.65, stdev=1539.31
    clat percentiles (usec):
     |  1.00th=[ 4047],  5.00th=[ 4621], 10.00th=[ 4948], 20.00th=[ 5276],
     | 30.00th=[ 5473], 40.00th=[ 5669], 50.00th=[ 5735], 60.00th=[ 5866],
     | 70.00th=[ 5997], 80.00th=[ 6194], 90.00th=[ 6652], 95.00th=[ 7767],
     | 99.00th=[14091], 99.50th=[15270], 99.90th=[17957], 99.95th=[24249],
     | 99.99th=[30016]
   bw (  MiB/s): min= 4552, max= 5058, per=100.00%, avg=4818.99, stdev=119.09, samples=120
   iops        : min= 1138, max= 1264, avg=1204.55, stdev=29.75, samples=120
  lat (usec)   : 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%, 4=0.74%, 10=96.68%, 20=2.50%, 50=0.06%
  cpu          : usr=41.67%, sys=39.12%, ctx=46501, majf=0, minf=8232
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,72222,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=4815MiB/s (5049MB/s), 4815MiB/s-4815MiB/s (5049MB/s-5049MB/s), io=282GiB (303GB), run=60001-60001msec

Disk stats (read/write):
    md0: ios=0/2305321, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/462222, aggrmerge=0/7, aggrticks=0/809132, aggrin_queue=763212, aggrutil=78.44%
  nvme0n1: ios=0/462225, merge=0/6, ticks=0/723764, in_queue=678840, util=74.29%
  nvme3n1: ios=0/462220, merge=0/6, ticks=0/867552, in_queue=821964, util=78.06%
  nvme2n1: ios=0/462220, merge=0/6, ticks=0/652296, in_queue=604092, util=72.10%
  nvme1n1: ios=0/462225, merge=0/6, ticks=0/1077684, in_queue=1036168, util=78.44%
  nvme4n1: ios=0/462223, merge=0/11, ticks=0/724364, in_queue=675000, util=73.74%

2.6 4M 顺序读

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4m_seq_read \
  --rw=read \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
Jobs: 1 (f=1): [R(1)][100.0%][r=13.4GiB/s,w=0KiB/s][r=3430,w=0 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=17975
   read: IOPS=3385, BW=13.2GiB/s (14.2GB/s)(794GiB/60002msec)
    slat (usec): min=157, max=25717, avg=287.34, stdev=116.19
    clat (usec): min=543, max=28567, avg=2073.83, stdev=510.64
     lat (usec): min=743, max=29028, avg=2361.44, stdev=559.20
    clat percentiles (usec):
     |  1.00th=[ 1336],  5.00th=[ 1516], 10.00th=[ 1614], 20.00th=[ 1713],
     | 30.00th=[ 1795], 40.00th=[ 1893], 50.00th=[ 1975], 60.00th=[ 2057],
     | 70.00th=[ 2180], 80.00th=[ 2343], 90.00th=[ 2704], 95.00th=[ 3032],
     | 99.00th=[ 3720], 99.50th=[ 3982], 99.90th=[ 5080], 99.95th=[ 5735],
     | 99.99th=[ 7504]
   bw (  MiB/s): min=11046, max=15038, per=100.00%, avg=13554.51, stdev=812.98, samples=120
   iops        : min= 2761, max= 3759, avg=3388.38, stdev=203.25, samples=120
  lat (usec)   : 750=0.01%, 1000=0.04%
  lat (msec)   : 2=52.20%, 4=47.29%, 10=0.46%, 20=0.01%, 50=0.01%
  cpu          : usr=0.81%, sys=97.04%, ctx=9999, majf=0, minf=8211
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=203138,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=13.2GiB/s (14.2GB/s), 13.2GiB/s-13.2GiB/s (14.2GB/s-14.2GB/s), io=794GiB (852GB), run=60002-60002msec

Disk stats (read/write):
    md0: ios=7321063/16, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1467827/2, aggrmerge=9/1, aggrticks=1054376/0, aggrin_queue=962353, aggrutil=99.57%
  nvme0n1: ios=1467825/6, merge=0/4, ticks=1057288/0, in_queue=966596, util=99.57%
  nvme3n1: ios=1467745/2, merge=16/1, ticks=1063604/0, in_queue=971024, util=99.50%
  nvme2n1: ios=1467840/0, merge=0/0, ticks=1050392/0, in_queue=958220, util=99.53%
  nvme1n1: ios=1467885/2, merge=16/0, ticks=1042768/0, in_queue=951248, util=99.53%
  nvme4n1: ios=1467840/1, merge=16/0, ticks=1057832/0, in_queue=964680, util=99.50%

3. RAID 1

3.1 创建 RAID 1

查看磁盘

1
lsblk -d -o NAME,SIZE,TYPE | grep nvme

1
2
3
4
5
6
nvme0n1 745.2G disk
nvme1n1 745.2G disk
nvme2n1 745.2G disk
nvme3n1 745.2G disk
nvme4n1 745.2G disk
nvme5n1 745.2G disk

停掉 md0

1
2
umount /mnt/mdadm
mdadm --stop /dev/md0

创建 RAID

1
mdadm --create --verbose /dev/md0 --level=1 --raid-devices=5 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1

这里的 level=1 表示 RAID 1，raid-devices=5 表示使用 5 块盘。

查看 RAID 状态

1
mdadm --detail /dev/md0

等盘 State : clean 再操作。

格式化 RAID

1
mkfs.xfs -f /dev/md0

挂载 RAID

1
2
mkdir -p /mnt/mdadm
mount /dev/md0 /mnt/mdadm

查看磁盘大小

1
df -h /mnt/mdadm/

1
2
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        745G  5.3G  740G   1% /mnt/mdadm

3.2 4k 随机写

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
fio \
  --name=4k_randwrite \
  --rw=randwrite \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randwrite.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][w=667MiB/s][w=171k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=2652550
  write: IOPS=82.4k, BW=322MiB/s (338MB/s)(37.7GiB/120002msec); 0 zone resets
    slat (usec): min=9, max=97750, avg=89.03, stdev=302.57
    clat (nsec): min=1173, max=113435k, avg=3013092.85, stdev=2295088.76
     lat (usec): min=31, max=113694, avg=3102.50, stdev=2341.60
    clat percentiles (usec):
     |  1.00th=[  955],  5.00th=[ 1254], 10.00th=[ 1418], 20.00th=[ 1647],
     | 30.00th=[ 1860], 40.00th=[ 2089], 50.00th=[ 2376], 60.00th=[ 2737],
     | 70.00th=[ 3261], 80.00th=[ 4015], 90.00th=[ 5211], 95.00th=[ 6456],
     | 99.00th=[11207], 99.50th=[15008], 99.90th=[23462], 99.95th=[28705],
     | 99.99th=[48497]
   bw (  KiB/s): min=115696, max=692616, per=99.75%, avg=328859.77, stdev=18064.76, samples=1912
   iops        : min=28924, max=173154, avg=82214.49, stdev=4516.17, samples=1912
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.02%, 750=0.20%, 1000=1.02%
  lat (msec)   : 2=35.14%, 4=43.29%, 10=18.99%, 20=1.14%, 50=0.18%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=5.14%, sys=25.04%, ctx=8249593, majf=0, minf=8356
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,9890927,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=322MiB/s (338MB/s), 322MiB/s-322MiB/s (338MB/s-338MB/s), io=37.7GiB (40.5GB), run=120002-120002msec

Disk stats (read/write):
    md0: ios=0/10261391, merge=0/0, ticks=0/3865644, in_queue=3865644, util=98.87%, aggrios=37784/10387166, aggrmerge=37454/191430, aggrticks=5046/1061219, aggrin_queue=1066265, aggrutil=99.44%
  nvme0n1: ios=188924/10236122, merge=187272/41518, ticks=25231/402344, in_queue=427575, util=99.41%
  nvme3n1: ios=0/10425016, merge=0/228820, ticks=0/990071, in_queue=990071, util=99.43%
  nvme2n1: ios=0/10425311, merge=0/228525, ticks=0/1238220, in_queue=1238219, util=99.43%
  nvme1n1: ios=0/10425217, merge=0/228619, ticks=0/1453918, in_queue=1453919, util=99.42%
  nvme4n1: ios=0/10424164, merge=0/229672, ticks=0/1221545, in_queue=1221545, util=99.44%

3.3 4k 随机读

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4k_randread \
  --rw=randread \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --filename=/mnt/mdadm/fio-4k-randread.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=0): [f(8)][100.0%][r=3149MiB/s][r=806k IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=2683004
  read: IOPS=734k, BW=2869MiB/s (3008MB/s)(336GiB/120001msec)
    slat (usec): min=2, max=9386, avg= 7.80, stdev= 5.01
    clat (usec): min=30, max=11243, avg=339.88, stdev=237.70
     lat (usec): min=34, max=11250, avg=347.77, stdev=237.83
    clat percentiles (usec):
     |  1.00th=[  112],  5.00th=[  174], 10.00th=[  200], 20.00th=[  235],
     | 30.00th=[  255], 40.00th=[  273], 50.00th=[  289], 60.00th=[  310],
     | 70.00th=[  338], 80.00th=[  375], 90.00th=[  457], 95.00th=[  652],
     | 99.00th=[ 1483], 99.50th=[ 1811], 99.90th=[ 2606], 99.95th=[ 2999],
     | 99.99th=[ 3949]
   bw (  MiB/s): min= 1674, max= 4300, per=100.00%, avg=2869.67, stdev=81.34, samples=1912
   iops        : min=428666, max=1100852, avg=734634.40, stdev=20824.14, samples=1912
  lat (usec)   : 50=0.01%, 100=0.53%, 250=26.95%, 500=64.53%, 750=3.90%
  lat (usec)   : 1000=1.54%
  lat (msec)   : 2=2.19%, 4=0.33%, 10=0.01%, 20=0.01%
  cpu          : usr=19.84%, sys=70.85%, ctx=4603551, majf=0, minf=7479
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=88136288,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=2869MiB/s (3008MB/s), 2869MiB/s-2869MiB/s (3008MB/s-3008MB/s), io=336GiB (361GB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=87946164/6, merge=0/0, ticks=17355860/0, in_queue=17355860, util=100.00%, aggrios=17665861/154492, aggrmerge=37864/151457, aggrticks=3493488/18341, aggrin_queue=3511830, aggrutil=100.00%
  nvme0n1: ios=30093595/79, merge=189323/0, ticks=8458371/1, in_queue=8458373, util=100.00%
  nvme3n1: ios=14502719/193095, merge=0/189324, ticks=2242312/21979, in_queue=2264292, util=87.26%
  nvme2n1: ios=14586400/193093, merge=0/189323, ticks=2262548/21228, in_queue=2283777, util=87.26%
  nvme1n1: ios=14741596/193106, merge=0/189321, ticks=2284985/22290, in_queue=2307275, util=87.26%
  nvme4n1: ios=14404995/193090, merge=0/189320, ticks=2219228/26209, in_queue=2245436, util=87.26%

3.4 4k 随机读写

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
fio \
  --name=4k_randrw \
  --rw=randrw \
  --rwmixread=70 \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=1 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randrw.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=247MiB/s,w=105MiB/s][r=63.2k,w=26.9k IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=2707685
  read: IOPS=57.6k, BW=225MiB/s (236MB/s)(26.4GiB/120001msec)
    slat (usec): min=3, max=8035, avg= 7.60, stdev=10.20
    clat (nsec): min=920, max=16185k, avg=107158.69, stdev=52548.88
     lat (usec): min=16, max=16192, avg=114.84, stdev=53.93
    clat percentiles (usec):
     |  1.00th=[   83],  5.00th=[   84], 10.00th=[   85], 20.00th=[   87],
     | 30.00th=[   88], 40.00th=[   90], 50.00th=[   97], 60.00th=[  102],
     | 70.00th=[  105], 80.00th=[  128], 90.00th=[  149], 95.00th=[  163],
     | 99.00th=[  229], 99.50th=[  258], 99.90th=[  302], 99.95th=[  396],
     | 99.99th=[ 2311]
   bw (  KiB/s): min=179614, max=263334, per=100.00%, avg=230318.28, stdev=3146.19, samples=1912
   iops        : min=44903, max=65833, avg=57579.30, stdev=786.56, samples=1912
  write: IOPS=24.7k, BW=96.4MiB/s (101MB/s)(11.3GiB/120001msec); 0 zone resets
    slat (usec): min=8, max=6734, avg=21.99, stdev=16.46
    clat (nsec): min=679, max=22236k, avg=29493.50, stdev=96588.38
     lat (usec): min=21, max=22264, avg=51.86, stdev=97.88
    clat percentiles (nsec):
     |  1.00th=[   1256],  5.00th=[   5216], 10.00th=[  13120],
     | 20.00th=[  14912], 30.00th=[  15808], 40.00th=[  16768],
     | 50.00th=[  17536], 60.00th=[  18560], 70.00th=[  20096],
     | 80.00th=[  23424], 90.00th=[  45824], 95.00th=[  92672],
     | 99.00th=[ 264192], 99.50th=[ 358400], 99.90th=[ 577536],
     | 99.95th=[ 765952], 99.99th=[4046848]
   bw (  KiB/s): min=75695, max=117144, per=100.00%, avg=98725.13, stdev=1379.97, samples=1912
   iops        : min=18923, max=29286, avg=24680.95, stdev=345.02, samples=1912
  lat (nsec)   : 750=0.01%, 1000=0.08%
  lat (usec)   : 2=1.09%, 4=0.25%, 10=1.13%, 20=18.19%, 50=6.41%
  lat (usec)   : 100=38.67%, 250=33.44%, 500=0.67%, 750=0.04%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=4.14%, sys=14.80%, ctx=9901027, majf=0, minf=7895
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=6908303,2961139,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=26.4GiB (28.3GB), run=120001-120001msec
  WRITE: bw=96.4MiB/s (101MB/s), 96.4MiB/s-96.4MiB/s (101MB/s-101MB/s), io=11.3GiB (12.1GB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=6907167/2960677, merge=0/0, ticks=712932/107392, in_queue=820324, util=100.00%, aggrios=1422070/3123074, aggrmerge=40251/161015, aggrticks=150936/102939, aggrin_queue=253875, aggrutil=100.00%
  nvme0n1: ios=2815994/2961442, merge=201255/4, ticks=284862/50206, in_queue=335068, util=100.00%
  nvme3n1: ios=885144/3163482, merge=0/201268, ticks=96499/112731, in_queue=209229, util=100.00%
  nvme2n1: ios=1124500/3163482, merge=0/201268, ticks=123379/122800, in_queue=246179, util=100.00%
  nvme1n1: ios=1496981/3163482, merge=0/201268, ticks=164100/104848, in_queue=268948, util=100.00%
  nvme4n1: ios=787733/3163482, merge=0/201268, ticks=85842/124112, in_queue=209954, util=100.00%

3.5 4M 顺序写

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4m_seq_write \
  --rw=write \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=977MiB/s][w=244 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=2739437
  write: IOPS=242, BW=969MiB/s (1016MB/s)(56.8GiB/60029msec); 0 zone resets
    slat (usec): min=1240, max=40022, avg=3601.34, stdev=6282.34
    clat (usec): min=6126, max=79244, avg=29402.88, stdev=15252.97
     lat (usec): min=11895, max=81106, avg=33005.52, stdev=14984.86
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[12256], 10.00th=[12911], 20.00th=[14353],
     | 30.00th=[16057], 40.00th=[18220], 50.00th=[22152], 60.00th=[36963],
     | 70.00th=[43779], 80.00th=[46924], 90.00th=[49546], 95.00th=[52167],
     | 99.00th=[56886], 99.50th=[58983], 99.90th=[66847], 99.95th=[70779],
     | 99.99th=[76022]
   bw (  KiB/s): min=925696, max=1048576, per=100.00%, avg=993005.50, stdev=32797.75, samples=119
   iops        : min=  226, max=  256, avg=242.43, stdev= 8.01, samples=119
  lat (msec)   : 10=0.01%, 20=45.48%, 50=45.13%, 100=9.38%
  cpu          : usr=8.75%, sys=40.43%, ctx=9015, majf=0, minf=9275
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,14543,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=969MiB/s (1016MB/s), 969MiB/s-969MiB/s (1016MB/s-1016MB/s), io=56.8GiB (61.0GB), run=60029-60029msec

Disk stats (read/write):
    md0: ios=0/466632, merge=0/0, ticks=0/9388192, in_queue=9388192, util=100.00%, aggrios=2530/478947, aggrmerge=2509/10047, aggrticks=738/6505043, aggrin_queue=6505781, aggrutil=99.54%
  nvme0n1: ios=12650/468830, merge=12549/7, ticks=3691/5552350, in_queue=5556041, util=98.32%
  nvme3n1: ios=0/481465, merge=0/12557, ticks=0/6885075, in_queue=6885075, util=99.54%
  nvme2n1: ios=0/481480, merge=0/12557, ticks=0/6705007, in_queue=6705007, util=99.31%
  nvme1n1: ios=0/481480, merge=0/12558, ticks=0/6724055, in_queue=6724055, util=99.37%
  nvme4n1: ios=0/481481, merge=0/12558, ticks=0/6658729, in_queue=6658729, util=99.28%

3.6 4M 顺序读

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4m_seq_read \
  --rw=read \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=2991MiB/s][r=747 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=2764172
  read: IOPS=762, BW=3049MiB/s (3197MB/s)(179GiB/60005msec)
    slat (usec): min=249, max=3163, avg=530.66, stdev=184.39
    clat (usec): min=768, max=21504, avg=9960.66, stdev=1419.33
     lat (usec): min=1591, max=21939, avg=10491.80, stdev=1394.86
    clat percentiles (usec):
     |  1.00th=[ 3326],  5.00th=[ 8029], 10.00th=[ 8848], 20.00th=[ 9372],
     | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159], 60.00th=[10290],
     | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207], 95.00th=[11600],
     | 99.00th=[12518], 99.50th=[13042], 99.90th=[14222], 99.95th=[15008],
     | 99.99th=[18482]
   bw (  MiB/s): min= 2810, max= 3944, per=100.00%, avg=3050.60, stdev=212.52, samples=119
   iops        : min=  702, max=  986, avg=762.57, stdev=53.14, samples=119
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.39%, 4=0.92%, 10=42.17%, 20=56.51%, 50=0.01%
  cpu          : usr=0.54%, sys=41.41%, ctx=41948, majf=0, minf=8209
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=45733,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=3049MiB/s (3197MB/s), 3049MiB/s-3049MiB/s (3197MB/s-3197MB/s), io=179GiB (192GB), run=60005-60005msec

Disk stats (read/write):
    md0: ios=1489199/2, merge=0/0, ticks=10292636/0, in_queue=10292636, util=100.00%, aggrios=299771/9965, aggrmerge=2993/9768, aggrticks=2063822/2188, aggrin_queue=2066011, aggrutil=100.00%
  nvme0n1: ios=1438808/39, merge=14867/0, ticks=9987926/12, in_queue=9987937, util=100.00%
  nvme3n1: ios=0/12445, merge=0/12211, ticks=0/2339, in_queue=2339, util=59.79%
  nvme2n1: ios=1041/12437, merge=2/12212, ticks=5788/2668, in_queue=8456, util=59.73%
  nvme1n1: ios=58368/12458, merge=95/12211, ticks=322095/3827, in_queue=325922, util=60.20%
  nvme4n1: ios=640/12446, merge=2/12210, ticks=3305/2097, in_queue=5402, util=60.14%

3.7 4M 多进程读

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4m_seq_read \
  --rw=read \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=4 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
Starting 4 processes
4m_seq_read: Laying out IO file (1 file / 51200MiB)
4m_seq_read: Laying out IO file (1 file / 51200MiB)
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 4 (f=4): [R(4)][100.0%][r=6671MiB/s][r=1667 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=4): err= 0: pid=2820380
  read: IOPS=1729, BW=6919MiB/s (7255MB/s)(406GiB/60024msec)
    slat (usec): min=193, max=3597, avg=509.02, stdev=173.82
    clat (usec): min=728, max=135913, avg=17982.17, stdev=12712.06
     lat (usec): min=1474, max=136601, avg=18491.66, stdev=12703.65
    clat percentiles (usec):
     |  1.00th=[  1778],  5.00th=[  4490], 10.00th=[  6915], 20.00th=[  9110],
     | 30.00th=[  9765], 40.00th=[ 10290], 50.00th=[ 11207], 60.00th=[ 16319],
     | 70.00th=[ 22414], 80.00th=[ 30016], 90.00th=[ 36963], 95.00th=[ 42730],
     | 99.00th=[ 54789], 99.50th=[ 60556], 99.90th=[ 73925], 99.95th=[ 80217],
     | 99.99th=[105382]
   bw (  MiB/s): min= 2728, max=11256, per=100.00%, avg=6927.51, stdev=618.57, samples=476
   iops        : min=  682, max= 2814, avg=1731.83, stdev=154.64, samples=476
  lat (usec)   : 750=0.01%, 1000=0.02%
  lat (msec)   : 2=1.33%, 4=2.84%, 10=29.61%, 20=32.01%, 50=32.26%
  lat (msec)   : 100=1.91%, 250=0.01%
  cpu          : usr=0.33%, sys=22.52%, ctx=90244, majf=0, minf=38564
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=103825,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=6919MiB/s (7255MB/s), 6919MiB/s-6919MiB/s (7255MB/s-7255MB/s), io=406GiB (435GB), run=60024-60024msec

Disk stats (read/write):
    md0: ios=3589357/2, merge=0/0, ticks=50536872/24, in_queue=50536896, util=100.00%, aggrios=720567/7054, aggrmerge=1793/6945, aggrticks=10133686/1979, aggrin_queue=10135665, aggrutil=100.00%
  nvme0n1: ios=1595779/42, merge=8724/0, ticks=37877166/106, in_queue=37877272, util=100.00%
  nvme3n1: ios=137295/8808, merge=23/8683, ticks=811084/1485, in_queue=812569, util=55.37%
  nvme2n1: ios=781918/8808, merge=105/8681, ticks=5042680/2424, in_queue=5045103, util=77.49%
  nvme1n1: ios=953641/8814, merge=104/8682, ticks=6080949/3472, in_queue=6084421, util=85.15%
  nvme4n1: ios=134204/8798, merge=9/8682, ticks=856553/2409, in_queue=858961, util=54.78%

RAID 1 下多线程比单线程，大文件读性能提升明显。

4. RAID 5

4.1 创建 RAID 5

查看磁盘

1
lsblk -d -o NAME,SIZE,TYPE | grep nvme

1
2
3
4
5
6
nvme0n1 745.2G disk
nvme1n1 745.2G disk
nvme2n1 745.2G disk
nvme3n1 745.2G disk
nvme4n1 745.2G disk
nvme5n1 745.2G disk

停掉 md0

1
2
umount /mnt/mdadm
mdadm --stop /dev/md0

创建 RAID

1
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=5 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1

查看 RAID 状态

1
mdadm --detail /dev/md0

等盘 State : clean 再操作。

格式化 RAID

1
mkfs.xfs -f /dev/md0

挂载 RAID

1
2
mkdir -p /mnt/mdadm
mount /dev/md0 /mnt/mdadm

查看磁盘大小

1
df -h /mnt/mdadm/

1
2
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        3.0T  3.1G  3.0T   1% /mnt/mdadm

4.2 4k 随机写

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
fio \
  --name=4k_randwrite \
  --rw=randwrite \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randwrite.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][r=0KiB/s,w=128MiB/s][r=0,w=32.7k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=51252
  write: IOPS=29.4k, BW=115MiB/s (120MB/s)(13.4GiB/120011msec)
    slat (usec): min=4, max=9439.1k, avg=28.79, stdev=7258.31
    clat (usec): min=161, max=16550k, avg=8685.16, stdev=66489.70
     lat (usec): min=184, max=16550k, avg=8714.07, stdev=66884.47
    clat percentiles (msec):
     |  1.00th=[    5],  5.00th=[    6], 10.00th=[    6], 20.00th=[    7],
     | 30.00th=[    7], 40.00th=[    8], 50.00th=[    8], 60.00th=[    9],
     | 70.00th=[    9], 80.00th=[   10], 90.00th=[   11], 95.00th=[   12],
     | 99.00th=[   14], 99.50th=[   18], 99.90th=[   55], 99.95th=[   66],
     | 99.99th=[  584]
   bw (  KiB/s): min= 6236, max=27078, per=12.81%, avg=15042.38, stdev=2559.85, samples=1876
   iops        : min= 1559, max= 6769, avg=3760.36, stdev=639.97, samples=1876
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.22%, 10=85.41%, 20=13.89%, 50=0.30%
  lat (msec)   : 100=0.12%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2000=0.01%, >=2000=0.01%
  cpu          : usr=1.14%, sys=8.79%, ctx=2975367, majf=0, minf=5335
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,3523217,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=115MiB/s (120MB/s), 115MiB/s-115MiB/s (120MB/s-120MB/s), io=13.4GiB (14.4GB), run=120011-120011msec

Disk stats (read/write):
    md0: ios=1/3524290, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1431863/1419187, aggrmerge=437534/125725, aggrticks=207352/66853, aggrin_queue=100468, aggrutil=31.43%
  nvme0n1: ios=1438642/1412948, merge=545900/19325, ticks=206960/65304, in_queue=99152, util=30.91%
  nvme3n1: ios=1436346/1410694, merge=545590/19081, ticks=208604/64388, in_queue=98924, util=31.31%
  nvme2n1: ios=1437739/1411528, merge=548042/19440, ticks=207920/65316, in_queue=99248, util=31.07%
  nvme1n1: ios=1438882/1412818, merge=548132/19412, ticks=208776/65200, in_queue=100252, util=31.43%
  nvme4n1: ios=1407707/1447950, merge=6/551371, ticks=204500/74060, in_queue=104764, util=30.68%

4.3 4k 随机读

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4k_randread \
  --rw=randread \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --filename=/mnt/mdadm/fio-4k-randread.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [r(8)][39.7%][r=457MiB/s][r=117k IOPS][eta 01m:13s]
Jobs: 8 (f=8): [r(8)][100.0%][r=430MiB/s][r=110k IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=217473
  read: IOPS=112k, BW=437MiB/s (458MB/s)(51.2GiB/120003msec)
    slat (usec): min=2, max=17923, avg= 9.85, stdev=18.68
    clat (usec): min=185, max=6460.9k, avg=2278.74, stdev=1852.82
     lat (usec): min=197, max=6460.9k, avg=2288.70, stdev=1852.94
    clat percentiles (usec):
     |  1.00th=[ 1778],  5.00th=[ 1909], 10.00th=[ 1975], 20.00th=[ 2057],
     | 30.00th=[ 2114], 40.00th=[ 2180], 50.00th=[ 2245], 60.00th=[ 2278],
     | 70.00th=[ 2343], 80.00th=[ 2442], 90.00th=[ 2573], 95.00th=[ 2704],
     | 99.00th=[ 3326], 99.50th=[ 3818], 99.90th=[ 6652], 99.95th=[12125],
     | 99.99th=[23987]
   bw (  KiB/s): min=392690, max=505688, per=100.00%, avg=447417.35, stdev=2872.96, samples=1912
   iops        : min=98172, max=126422, avg=111854.24, stdev=718.25, samples=1912
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.04%
  lat (msec)   : 2=12.52%, 4=87.02%, 10=0.34%, 20=0.04%, 50=0.02%
  lat (msec)   : >=2000=0.01%
  cpu          : usr=4.92%, sys=17.20%, ctx=8592874, majf=0, minf=19666
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=13411036,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=437MiB/s (458MB/s), 437MiB/s-437MiB/s (458MB/s-458MB/s), io=51.2GiB (54.9GB), run=120003-120003msec

Disk stats (read/write):
    md0: ios=13409323/13, merge=0/0, ticks=30340800/82756, in_queue=30423556, util=100.00%, aggrios=2705594/6089, aggrmerge=675495/168948, aggrticks=807940/2953, aggrin_queue=810893, aggrutil=100.00%
  nvme0n1: ios=2710474/77, merge=845726/0, ticks=781487/2, in_queue=781490, util=99.94%
  nvme3n1: ios=2712213/73, merge=842766/21, ticks=803642/3, in_queue=803645, util=99.94%
  nvme2n1: ios=2710710/73, merge=843898/21, ticks=845539/4, in_queue=845542, util=99.95%
  nvme1n1: ios=2715919/76, merge=845043/2, ticks=793915/5, in_queue=793921, util=99.94%
  nvme4n1: ios=2678658/30147, merge=42/844698, ticks=815117/14751, in_queue=829868, util=100.00%

4.4 4k 随机读写

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
fio \
  --name=4k_randrw \
  --rw=randrw \
  --rwmixread=70 \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=1 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randrw.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=87.4MiB/s,w=37.9MiB/s][r=22.4k,w=9693 IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=7507
   read: IOPS=17.1k, BW=66.9MiB/s (70.2MB/s)(8031MiB/120002msec)
    slat (usec): min=3, max=62397, avg=12.54, stdev=221.94
    clat (nsec): min=882, max=777393k, avg=251848.42, stdev=1742140.37
     lat (usec): min=10, max=777402, avg=264.55, stdev=1759.85
    clat percentiles (usec):
     |  1.00th=[  101],  5.00th=[  112], 10.00th=[  122], 20.00th=[  139],
     | 30.00th=[  157], 40.00th=[  178], 50.00th=[  196], 60.00th=[  219],
     | 70.00th=[  247], 80.00th=[  285], 90.00th=[  355], 95.00th=[  449],
     | 99.00th=[ 1045], 99.50th=[ 1680], 99.90th=[ 4686], 99.95th=[ 6718],
     | 99.99th=[21365]
   bw (  KiB/s): min= 2621, max=18076, per=12.62%, avg=8648.18, stdev=2379.10, samples=1904
   iops        : min=  655, max= 4519, avg=2161.83, stdev=594.79, samples=1904
  write: IOPS=7353, BW=28.7MiB/s (30.1MB/s)(3447MiB/120002msec)
    slat (usec): min=5, max=62333, avg=17.07, stdev=197.56
    clat (nsec): min=993, max=778193k, avg=448717.35, stdev=2296947.52
     lat (usec): min=37, max=778226, avg=465.97, stdev=2308.84
    clat percentiles (usec):
     |  1.00th=[  133],  5.00th=[  155], 10.00th=[  178], 20.00th=[  227],
     | 30.00th=[  265], 40.00th=[  306], 50.00th=[  347], 60.00th=[  388],
     | 70.00th=[  441], 80.00th=[  510], 90.00th=[  644], 95.00th=[  840],
     | 99.00th=[ 2180], 99.50th=[ 3523], 99.90th=[ 9241], 99.95th=[13435],
     | 99.99th=[39060]
   bw (  KiB/s): min= 1058, max= 7776, per=12.62%, avg=3711.53, stdev=1019.45, samples=1904
   iops        : min=  264, max= 1944, avg=927.59, stdev=254.87, samples=1904
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.57%, 250=56.65%, 500=33.73%, 750=5.85%, 1000=1.37%
  lat (msec)   : 2=1.19%, 4=0.40%, 10=0.17%, 20=0.03%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  cpu          : usr=1.17%, sys=5.26%, ctx=3049332, majf=0, minf=1963
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=2056053,882433,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=66.9MiB/s (70.2MB/s), 66.9MiB/s-66.9MiB/s (70.2MB/s-70.2MB/s), io=8031MiB (8422MB), run=120002-120002msec
  WRITE: bw=28.7MiB/s (30.1MB/s), 28.7MiB/s-28.7MiB/s (30.1MB/s-30.1MB/s), io=3447MiB (3614MB), run=120002-120002msec

Disk stats (read/write):
    md0: ios=2054042/881576, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1038923/407548, aggrmerge=4483248/1134718, aggrticks=137619/25332, aggrin_queue=42784, aggrutil=29.15%
  nvme0n1: ios=1107610/353099, merge=5604199/57, ticks=143444/11540, in_queue=37508, util=21.53%
  nvme3n1: ios=1107306/352408, merge=5603885/1, ticks=145064/11652, in_queue=37192, util=21.47%
  nvme2n1: ios=1107900/353371, merge=5604316/53, ticks=144164/11984, in_queue=37304, util=21.78%
  nvme1n1: ios=1107403/353443, merge=5603844/0, ticks=144516/11524, in_queue=36816, util=21.47%
  nvme4n1: ios=764398/625423, merge=0/5673479, ticks=110908/79960, in_queue=65104, util=29.15%

4.5 4M 顺序写

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4m_seq_write \
  --rw=write \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=905MiB/s][r=0,w=226 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=35663
  write: IOPS=233, BW=932MiB/s (978MB/s)(54.6GiB/60014msec)
    slat (usec): min=915, max=46600, avg=1949.51, stdev=657.29
    clat (msec): min=14, max=490, avg=32.34, stdev=13.20
     lat (msec): min=17, max=492, avg=34.29, stdev=13.21
    clat percentiles (msec):
     |  1.00th=[   22],  5.00th=[   24], 10.00th=[   26], 20.00th=[   28],
     | 30.00th=[   29], 40.00th=[   30], 50.00th=[   31], 60.00th=[   32],
     | 70.00th=[   34], 80.00th=[   36], 90.00th=[   40], 95.00th=[   44],
     | 99.00th=[   59], 99.50th=[   74], 99.90th=[  153], 99.95th=[  477],
     | 99.99th=[  489]
   bw (  KiB/s): min=90112, max=1223054, per=100.00%, avg=955317.09, stdev=107989.13, samples=120
   iops        : min=   22, max=  298, avg=233.02, stdev=26.34, samples=120
  lat (msec)   : 20=0.50%, 50=97.37%, 100=1.95%, 250=0.12%, 500=0.06%
  cpu          : usr=9.74%, sys=35.69%, ctx=8041, majf=0, minf=16447
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,13988,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=932MiB/s (978MB/s), 932MiB/s-932MiB/s (978MB/s-978MB/s), io=54.6GiB (58.7GB), run=60014-60014msec

Disk stats (read/write):
    md0: ios=0/446196, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=3056/113219, aggrmerge=90237/3484367, aggrticks=4905/76840, aggrin_queue=13444, aggrutil=12.88%
  nvme0n1: ios=3583/112577, merge=106272/3468500, ticks=6260/75552, in_queue=13344, util=12.63%
  nvme3n1: ios=3625/112582, merge=107246/3468845, ticks=5812/75836, in_queue=13468, util=12.73%
  nvme2n1: ios=3610/112597, merge=106945/3468757, ticks=5972/75664, in_queue=13340, util=12.70%
  nvme1n1: ios=3537/112593, merge=105151/3468752, ticks=6028/75708, in_queue=13396, util=12.78%
  nvme4n1: ios=925/115749, merge=25575/3546984, ticks=456/81440, in_queue=13676, util=12.88%

4.6 4M 顺序读

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4m_seq_read \
  --rw=read \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=0): [f(1)][100.0%][r=1224MiB/s][r=306 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=220920
  read: IOPS=317, BW=1271MiB/s (1333MB/s)(74.5GiB/60019msec)
    slat (usec): min=600, max=39534, avg=1207.89, stdev=363.00
    clat (usec): min=9972, max=57792, avg=23950.28, stdev=2393.21
     lat (usec): min=12659, max=61704, avg=25158.63, stdev=2424.67
    clat percentiles (usec):
     |  1.00th=[19268],  5.00th=[21103], 10.00th=[21890], 20.00th=[22414],
     | 30.00th=[22938], 40.00th=[23200], 50.00th=[23725], 60.00th=[23987],
     | 70.00th=[24511], 80.00th=[25035], 90.00th=[26346], 95.00th=[27657],
     | 99.00th=[32637], 99.50th=[34866], 99.90th=[43254], 99.95th=[48497],
     | 99.99th=[55837]
   bw (  MiB/s): min= 1176, max= 1524, per=100.00%, avg=1271.75, stdev=53.06, samples=119
   iops        : min=  294, max=  381, avg=317.92, stdev=13.25, samples=119
  lat (msec)   : 10=0.01%, 20=1.93%, 50=98.02%, 100=0.05%
  cpu          : usr=0.17%, sys=38.56%, ctx=19122, majf=0, minf=11636
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=19068,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=1271MiB/s (1333MB/s), 1271MiB/s-1271MiB/s (1333MB/s-1333MB/s), io=74.5GiB (80.0GB), run=60019-60019msec

Disk stats (read/write):
    md0: ios=597074/5, merge=0/0, ticks=13437000/92332, in_queue=13529332, util=100.00%, aggrios=124421/647, aggrmerge=3856308/18095, aggrticks=79087/835, aggrin_queue=79923, aggrutil=92.89%
  nvme0n1: ios=125015/30, merge=3874659/2, ticks=77880/5, in_queue=77885, util=92.55%
  nvme3n1: ios=125019/29, merge=3874638/0, ticks=79825/1, in_queue=79826, util=92.65%
  nvme2n1: ios=125014/30, merge=3874631/7, ticks=80032/2, in_queue=80034, util=92.37%
  nvme1n1: ios=125013/31, merge=3874507/2, ticks=78425/5, in_queue=78430, util=92.47%
  nvme4n1: ios=122045/3116, merge=3783108/90467, ticks=79277/4164, in_queue=83442, util=92.89%

5. RAID 6

5.1 创建 RAID 6

查看磁盘

1
lsblk -d -o NAME,SIZE,TYPE | grep nvme

1
2
3
4
5
6
nvme0n1 745.2G disk
nvme1n1 745.2G disk
nvme2n1 745.2G disk
nvme3n1 745.2G disk
nvme4n1 745.2G disk
nvme5n1 745.2G disk

停掉 md0

1
2
umount /mnt/mdadm
mdadm --stop /dev/md0

创建 RAID

1
mdadm --create --verbose /dev/md0 --level=6 --raid-devices=5 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1

查看 RAID 状态

1
mdadm --detail /dev/md0

等盘 State : clean 再操作。

格式化 RAID

1
mkfs.xfs -f /dev/md0

挂载 RAID

1
2
mkdir -p /mnt/mdadm
mount /dev/md0 /mnt/mdadm

查看磁盘大小

1
df -h /mnt/mdadm/

1
2
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        2.2T  2.3G  2.2T   1% /mnt/mdadm

5.2 4k 随机写

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
fio \
  --name=4k_randwrite \
  --rw=randwrite \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randwrite.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][r=0KiB/s,w=84.7MiB/s][r=0,w=21.7k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=32858
  write: IOPS=23.9k, BW=93.4MiB/s (97.9MB/s)(10.9GiB/120007msec)
    slat (usec): min=4, max=86282, avg=24.05, stdev=502.49
    clat (usec): min=182, max=601133, avg=10678.54, stdev=8101.29
     lat (usec): min=193, max=601159, avg=10702.73, stdev=8117.41
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[    9], 10.00th=[    9], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   10], 50.00th=[   10], 60.00th=[   11],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   14], 95.00th=[   15],
     | 99.00th=[   17], 99.50th=[   25], 99.90th=[   89], 99.95th=[  101],
     | 99.99th=[  338]
   bw (  KiB/s): min= 1539, max=14949, per=12.52%, avg=11970.40, stdev=1676.18, samples=1920
   iops        : min=  384, max= 3737, avg=2992.33, stdev=419.06, samples=1920
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.02%, 10=53.02%, 20=46.35%, 50=0.29%
  lat (msec)   : 100=0.26%, 250=0.02%, 500=0.02%, 750=0.01%
  cpu          : usr=1.02%, sys=7.14%, ctx=2528554, majf=0, minf=366
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,2868897,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=93.4MiB/s (97.9MB/s), 93.4MiB/s-93.4MiB/s (97.9MB/s-97.9MB/s), io=10.9GiB (11.8GB), run=120007-120007msec

Disk stats (read/write):
    md0: ios=1/2872252, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1165819/1739264, aggrmerge=382175/177222, aggrticks=172486/78324, aggrin_queue=79014, aggrutil=28.17%
  nvme0n1: ios=1166238/1738908, merge=382185/176783, ticks=172300/78104, in_queue=78904, util=27.67%
  nvme3n1: ios=1165951/1738966, merge=382025/177845, ticks=170916/77944, in_queue=76912, util=27.36%
  nvme2n1: ios=1165506/1739719, merge=382121/177330, ticks=171780/78812, in_queue=78568, util=27.66%
  nvme1n1: ios=1165078/1740080, merge=382450/176873, ticks=172332/78628, in_queue=79588, util=27.62%
  nvme4n1: ios=1166326/1738647, merge=382095/177282, ticks=175104/78132, in_queue=81100, util=28.17%

5.3 4k 随机读

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4k_randread \
  --rw=randread \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --filename=/mnt/mdadm/fio-4k-randread.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [r(8)][100.0%][r=3998MiB/s,w=0KiB/s][r=1023k,w=0 IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=32831
   read: IOPS=999k, BW=3902MiB/s (4091MB/s)(457GiB/120001msec)
    slat (usec): min=2, max=22645, avg= 6.15, stdev=10.62
    clat (usec): min=22, max=29350, avg=249.28, stdev=233.95
     lat (usec): min=30, max=29360, avg=255.50, stdev=234.46
    clat percentiles (usec):
     |  1.00th=[   92],  5.00th=[  110], 10.00th=[  125], 20.00th=[  151],
     | 30.00th=[  172], 40.00th=[  192], 50.00th=[  210], 60.00th=[  233],
     | 70.00th=[  258], 80.00th=[  293], 90.00th=[  355], 95.00th=[  424],
     | 99.00th=[ 1401], 99.50th=[ 1942], 99.90th=[ 2704], 99.95th=[ 2933],
     | 99.99th=[ 4080]
   bw (  KiB/s): min=312313, max=605791, per=12.52%, avg=500364.02, stdev=36037.23, samples=1920
   iops        : min=78078, max=151447, avg=125090.71, stdev=9009.29, samples=1920
  lat (usec)   : 50=0.01%, 100=2.41%, 250=65.06%, 500=29.42%, 750=1.29%
  lat (usec)   : 1000=0.36%
  lat (msec)   : 2=1.00%, 4=0.44%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=21.24%, sys=76.36%, ctx=2153127, majf=0, minf=4745
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=119863453,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=3902MiB/s (4091MB/s), 3902MiB/s-3902MiB/s (4091MB/s-4091MB/s), io=457GiB (491GB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=119748678/9, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=24049579/25441, aggrmerge=2245611/747233, aggrticks=3894218/4728, aggrin_queue=4242146, aggrutil=100.00%
  nvme0n1: ios=24051815/25428, merge=2245621/747293, ticks=3887344/4740, in_queue=4203696, util=100.00%
  nvme3n1: ios=24050100/25478, merge=2245619/747218, ticks=3891800/4728, in_queue=4184680, util=100.00%
  nvme2n1: ios=24047787/25409, merge=2245611/747216, ticks=3893712/4568, in_queue=4334888, util=100.00%
  nvme1n1: ios=24046232/25437, merge=2245619/747172, ticks=3884764/4656, in_queue=4179812, util=100.00%
  nvme4n1: ios=24051963/25454, merge=2245589/747267, ticks=3913472/4948, in_queue=4307656, util=100.00%

5.4 4k 随机读写

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
fio \
  --name=4k_randrw \
  --rw=randrw \
  --rwmixread=70 \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=1 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randrw.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=56.4MiB/s,w=24.1MiB/s][r=14.4k,w=6157 IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=14587
   read: IOPS=15.4k, BW=60.4MiB/s (63.3MB/s)(7242MiB/120002msec)
    slat (usec): min=2, max=6413, avg= 9.28, stdev=13.42
    clat (nsec): min=1423, max=11909k, avg=101092.62, stdev=45837.96
     lat (usec): min=17, max=11985, avg=110.52, stdev=48.00
    clat percentiles (usec):
     |  1.00th=[   86],  5.00th=[   87], 10.00th=[   88], 20.00th=[   90],
     | 30.00th=[   91], 40.00th=[   92], 50.00th=[   94], 60.00th=[  101],
     | 70.00th=[  105], 80.00th=[  108], 90.00th=[  114], 95.00th=[  130],
     | 99.00th=[  182], 99.50th=[  212], 99.90th=[  314], 99.95th=[  404],
     | 99.99th=[ 1565]
   bw (  KiB/s): min= 2528, max=10092, per=12.51%, avg=7733.86, stdev=938.41, samples=1918
   iops        : min=  632, max= 2523, avg=1933.25, stdev=234.62, samples=1918
  write: IOPS=6628, BW=25.9MiB/s (27.2MB/s)(3107MiB/120002msec)
    slat (usec): min=4, max=94806, avg=17.18, stdev=414.83
    clat (nsec): min=1210, max=317527k, avg=926268.22, stdev=2000717.09
     lat (usec): min=34, max=317579, avg=943.61, stdev=2046.66
    clat percentiles (usec):
     |  1.00th=[  131],  5.00th=[  157], 10.00th=[  198], 20.00th=[  375],
     | 30.00th=[  611], 40.00th=[  709], 50.00th=[  766], 60.00th=[  824],
     | 70.00th=[  938], 80.00th=[ 1139], 90.00th=[ 1549], 95.00th=[ 2147],
     | 99.00th=[ 4359], 99.50th=[ 5735], 99.90th=[11863], 99.95th=[17957],
     | 99.99th=[52167]
   bw (  KiB/s): min= 1138, max= 4080, per=12.51%, avg=3318.17, stdev=374.55, samples=1918
   iops        : min=  284, max= 1020, avg=829.25, stdev=93.58, samples=1918
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=41.58%, 250=32.59%, 500=3.14%, 750=6.69%, 1000=8.09%
  lat (msec)   : 2=6.14%, 4=1.38%, 10=0.33%, 20=0.03%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=1.08%, sys=4.22%, ctx=2730677, majf=0, minf=2468
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=1854030,795491,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=60.4MiB/s (63.3MB/s), 60.4MiB/s-60.4MiB/s (63.3MB/s-63.3MB/s), io=7242MiB (7594MB), run=120002-120002msec
  WRITE: bw=25.9MiB/s (27.2MB/s), 25.9MiB/s-25.9MiB/s (27.2MB/s-27.2MB/s), io=3107MiB (3258MB), run=120002-120002msec

Disk stats (read/write):
    md0: ios=1851592/794454, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=847523/477463, aggrmerge=3545063/33, aggrticks=123240/28927, aggrin_queue=35743, aggrutil=28.22%
  nvme0n1: ios=847037/477760, merge=3544990/50, ticks=117308/29140, in_queue=30700, util=17.38%
  nvme3n1: ios=847090/477951, merge=3545181/0, ticks=116560/29000, in_queue=29552, util=17.15%
  nvme2n1: ios=848126/477235, merge=3545137/54, ticks=116288/29228, in_queue=29844, util=17.01%
  nvme1n1: ios=847872/477076, merge=3544783/56, ticks=116780/28848, in_queue=30100, util=17.19%
  nvme4n1: ios=847490/477293, merge=3545226/9, ticks=149268/28420, in_queue=58520, util=28.22%

5.5 4M 顺序写

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4m_seq_write \
  --rw=write \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=788MiB/s][r=0,w=197 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=22155
  write: IOPS=195, BW=780MiB/s (818MB/s)(45.7GiB/60015msec)
    slat (usec): min=953, max=110735, avg=2128.11, stdev=1116.33
    clat (msec): min=13, max=725, avg=38.82, stdev=20.13
     lat (msec): min=16, max=727, avg=40.95, stdev=20.17
    clat percentiles (msec):
     |  1.00th=[   26],  5.00th=[   29], 10.00th=[   31], 20.00th=[   33],
     | 30.00th=[   34], 40.00th=[   35], 50.00th=[   37], 60.00th=[   39],
     | 70.00th=[   41], 80.00th=[   43], 90.00th=[   49], 95.00th=[   54],
     | 99.00th=[   70], 99.50th=[   79], 99.90th=[  257], 99.95th=[  701],
     | 99.99th=[  718]
   bw (  KiB/s): min=410420, max=952176, per=100.00%, avg=805716.88, stdev=69019.53, samples=119
   iops        : min=  100, max=  232, avg=196.55, stdev=16.85, samples=119
  lat (msec)   : 20=0.01%, 50=92.04%, 100=7.81%, 250=0.01%, 500=0.06%
  lat (msec)   : 750=0.07%
  cpu          : usr=8.52%, sys=32.93%, ctx=7255, majf=0, minf=25476
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,11703,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=780MiB/s (818MB/s), 780MiB/s-780MiB/s (818MB/s-818MB/s), io=45.7GiB (49.1GB), run=60015-60015msec

Disk stats (read/write):
    md0: ios=0/373778, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=3294/125707, aggrmerge=98650/3869303, aggrticks=5868/77138, aggrin_queue=14148, aggrutil=13.74%
  nvme0n1: ios=3345/125706, merge=100391/3869234, ticks=5944/77612, in_queue=14396, util=13.48%
  nvme3n1: ios=3297/125705, merge=98759/3869335, ticks=5616/77656, in_queue=14568, util=13.54%
  nvme2n1: ios=3250/125713, merge=96926/3869428, ticks=5856/77088, in_queue=14212, util=13.28%
  nvme1n1: ios=3263/125708, merge=97543/3869307, ticks=5860/77048, in_queue=13956, util=13.12%
  nvme4n1: ios=3315/125703, merge=99634/3869214, ticks=6064/76288, in_queue=13608, util=13.74%

5.6 4M 顺序读

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4m_seq_read \
  --rw=read \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=11.4GiB/s,w=0KiB/s][r=2926,w=0 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=16590
   read: IOPS=2725, BW=10.6GiB/s (11.4GB/s)(639GiB/60001msec)
    slat (usec): min=187, max=4728, avg=363.72, stdev=111.08
    clat (usec): min=452, max=14674, avg=2569.21, stdev=572.70
     lat (usec): min=891, max=17172, avg=2933.25, stdev=643.23
    clat percentiles (usec):
     |  1.00th=[ 1860],  5.00th=[ 2024], 10.00th=[ 2114], 20.00th=[ 2180],
     | 30.00th=[ 2278], 40.00th=[ 2343], 50.00th=[ 2409], 60.00th=[ 2507],
     | 70.00th=[ 2606], 80.00th=[ 2802], 90.00th=[ 3294], 95.00th=[ 3785],
     | 99.00th=[ 4686], 99.50th=[ 5080], 99.90th=[ 6390], 99.95th=[ 6915],
     | 99.99th=[ 8848]
   bw (  MiB/s): min= 6608, max=13373, per=100.00%, avg=10956.27, stdev=1081.21, samples=120
   iops        : min= 1652, max= 3343, avg=2738.76, stdev=270.18, samples=120
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=3.70%, 4=92.78%, 10=3.50%, 20=0.01%
  cpu          : usr=0.89%, sys=98.26%, ctx=5411, majf=0, minf=16432
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=163549,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=10.6GiB/s (11.4GB/s), 10.6GiB/s-10.6GiB/s (11.4GB/s-11.4GB/s), io=639GiB (686GB), run=60001-60001msec

Disk stats (read/write):
    md0: ios=6380450/5, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1850211/41, aggrmerge=1758800/7, aggrticks=762670/0, aggrin_queue=676558, aggrutil=99.62%
  nvme0n1: ios=1850105/42, merge=1758771/5, ticks=756424/0, in_queue=674672, util=99.25%
  nvme3n1: ios=1850214/41, merge=1758842/7, ticks=753412/0, in_queue=664152, util=99.18%
  nvme2n1: ios=1850318/42, merge=1758812/10, ticks=755052/0, in_queue=671332, util=99.26%
  nvme1n1: ios=1850167/42, merge=1758755/9, ticks=755708/0, in_queue=670380, util=99.36%
  nvme4n1: ios=1850255/42, merge=1758821/5, ticks=792756/0, in_queue=702256, util=99.62%

6. RAID 10

6.1 创建 RAID 10

查看磁盘

1
lsblk -d -o NAME,SIZE,TYPE | grep nvme

1
2
3
4
5
6
nvme0n1 745.2G disk
nvme1n1 745.2G disk
nvme2n1 745.2G disk
nvme3n1 745.2G disk
nvme4n1 745.2G disk
nvme5n1 745.2G disk

停掉 md0

1
2
umount /mnt/mdadm
mdadm --stop /dev/md0

创建 RAID

1
mdadm --create --verbose /dev/md0 --level=10 --raid-devices=5 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1

查看 RAID 状态

1
mdadm --detail /dev/md0

格式化 RAID

1
mkfs.xfs -f /dev/md0

挂载 RAID

1
2
mkdir -p /mnt/mdadm
mount /dev/md0 /mnt/mdadm

查看磁盘大小

1
df -h /mnt/mdadm/

1
2
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        1.9T   14G  1.9T   1% /mnt/mdadm

6.2 4k 随机写

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
fio \
  --name=4k_randwrite \
  --rw=randwrite \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randwrite.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][w=366MiB/s][w=93.7k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=2022537
  write: IOPS=61.3k, BW=239MiB/s (251MB/s)(28.1GiB/120001msec); 0 zone resets
    slat (usec): min=5, max=137150, avg=119.06, stdev=261.57
    clat (usec): min=13, max=144186, avg=4050.42, stdev=2085.72
     lat (usec): min=30, max=144397, avg=4169.79, stdev=2134.10
    clat percentiles (usec):
     |  1.00th=[ 1500],  5.00th=[ 1876], 10.00th=[ 2180], 20.00th=[ 2606],
     | 30.00th=[ 2933], 40.00th=[ 3261], 50.00th=[ 3621], 60.00th=[ 4047],
     | 70.00th=[ 4621], 80.00th=[ 5342], 90.00th=[ 6456], 95.00th=[ 7308],
     | 99.00th=[10159], 99.50th=[12256], 99.90th=[17957], 99.95th=[21103],
     | 99.99th=[36963]
   bw (  KiB/s): min=133296, max=483224, per=99.81%, avg=244684.63, stdev=10054.77, samples=1912
   iops        : min=33324, max=120806, avg=61171.05, stdev=2513.69, samples=1912
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.03%, 1000=0.07%
  lat (msec)   : 2=6.83%, 4=52.22%, 10=39.77%, 20=0.99%, 50=0.06%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=6.59%, sys=19.12%, ctx=6709292, majf=0, minf=17204
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,7354420,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=239MiB/s (251MB/s), 239MiB/s-239MiB/s (251MB/s-251MB/s), io=28.1GiB (30.1GB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=0/7533721, merge=0/0, ticks=0/236656, in_queue=236656, util=99.21%, aggrios=0/3007482, aggrmerge=0/12339, aggrticks=0/56780, aggrin_queue=56780, aggrutil=99.12%
  nvme0n1: ios=0/3007171, merge=0/12024, ticks=0/55734, in_queue=55734, util=99.10%
  nvme3n1: ios=0/3008403, merge=0/12566, ticks=0/56594, in_queue=56594, util=99.10%
  nvme2n1: ios=0/3009445, merge=0/12502, ticks=0/57499, in_queue=57499, util=99.12%
  nvme1n1: ios=0/3007620, merge=0/12321, ticks=0/55409, in_queue=55409, util=99.08%
  nvme4n1: ios=0/3004772, merge=0/12283, ticks=0/58667, in_queue=58667, util=99.09%

6.3 4k 随机读

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4k_randread \
  --rw=randread \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --filename=/mnt/mdadm/fio-4k-randread.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [r(8)][100.0%][r=3564MiB/s][r=912k IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=2056465
  read: IOPS=920k, BW=3594MiB/s (3769MB/s)(421GiB/120001msec)
    slat (usec): min=2, max=4722, avg= 7.04, stdev= 3.35
    clat (usec): min=67, max=6190, avg=270.45, stdev=51.39
     lat (usec): min=85, max=6196, avg=277.57, stdev=51.90
    clat percentiles (usec):
     |  1.00th=[  196],  5.00th=[  212], 10.00th=[  223], 20.00th=[  233],
     | 30.00th=[  241], 40.00th=[  251], 50.00th=[  260], 60.00th=[  273],
     | 70.00th=[  285], 80.00th=[  302], 90.00th=[  334], 95.00th=[  367],
     | 99.00th=[  445], 99.50th=[  482], 99.90th=[  586], 99.95th=[  660],
     | 99.99th=[  898]
   bw (  MiB/s): min= 3095, max= 4105, per=100.00%, avg=3596.72, stdev=21.24, samples=1912
   iops        : min=792330, max=1050942, avg=920759.45, stdev=5438.34, samples=1912
  lat (usec)   : 100=0.01%, 250=39.60%, 500=60.04%, 750=0.33%, 1000=0.02%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=21.35%, sys=78.63%, ctx=4412, majf=0, minf=4539
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=110420784,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=3594MiB/s (3769MB/s), 3594MiB/s-3594MiB/s (3769MB/s-3769MB/s), io=421GiB (452GB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=110375486/13, merge=0/0, ticks=12357644/0, in_queue=12357644, util=100.00%, aggrios=22084156/34, aggrmerge=0/0, aggrticks=2480023/1, aggrin_queue=2480024, aggrutil=100.00%
  nvme0n1: ios=22202499/40, merge=0/1, ticks=2478531/1, in_queue=2478532, util=100.00%
  nvme3n1: ios=22099756/32, merge=0/0, ticks=2480101/1, in_queue=2480101, util=100.00%
  nvme2n1: ios=22014556/32, merge=0/0, ticks=2481530/1, in_queue=2481531, util=100.00%
  nvme1n1: ios=22094397/37, merge=0/1, ticks=2480140/1, in_queue=2480141, util=100.00%
  nvme4n1: ios=22009576/33, merge=0/0, ticks=2479816/1, in_queue=2479816, util=100.00%

6.4 4k 随机读写

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
fio \
  --name=4k_randrw \
  --rw=randrw \
  --rwmixread=70 \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=1 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randrw.dat

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=241MiB/s,w=102MiB/s][r=61.8k,w=26.1k IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=2075206
  read: IOPS=62.8k, BW=245MiB/s (257MB/s)(28.7GiB/120001msec)
    slat (usec): min=3, max=4059, avg= 9.88, stdev= 5.02
    clat (nsec): min=921, max=9798.4k, avg=98826.21, stdev=34501.61
     lat (usec): min=16, max=9826, avg=108.83, stdev=35.18
    clat percentiles (usec):
     |  1.00th=[   83],  5.00th=[   85], 10.00th=[   86], 20.00th=[   88],
     | 30.00th=[   90], 40.00th=[   91], 50.00th=[   93], 60.00th=[   97],
     | 70.00th=[  104], 80.00th=[  106], 90.00th=[  111], 95.00th=[  133],
     | 99.00th=[  167], 99.50th=[  180], 99.90th=[  247], 99.95th=[  277],
     | 99.99th=[  807]
   bw (  KiB/s): min=228448, max=280636, per=100.00%, avg=251275.94, stdev=1185.49, samples=1912
   iops        : min=57112, max=70158, avg=62818.94, stdev=296.37, samples=1912
  write: IOPS=26.9k, BW=105MiB/s (110MB/s)(12.3GiB/120001msec); 0 zone resets
    slat (usec): min=5, max=3753, avg=14.48, stdev= 6.78
    clat (nsec): min=757, max=23490k, avg=23058.93, stdev=76226.61
     lat (usec): min=16, max=23506, avg=37.67, stdev=76.97
    clat percentiles (usec):
     |  1.00th=[   13],  5.00th=[   14], 10.00th=[   15], 20.00th=[   16],
     | 30.00th=[   19], 40.00th=[   22], 50.00th=[   23], 60.00th=[   24],
     | 70.00th=[   24], 80.00th=[   26], 90.00th=[   28], 95.00th=[   31],
     | 99.00th=[   43], 99.50th=[   52], 99.90th=[  124], 99.95th=[  289],
     | 99.99th=[ 3720]
   bw (  KiB/s): min=95504, max=123296, per=100.00%, avg=107703.38, stdev=609.52, samples=1912
   iops        : min=23876, max=30823, avg=26925.78, stdev=152.38, samples=1912
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=10.43%, 50=19.39%
  lat (usec)   : 100=43.70%, 250=26.37%, 500=0.06%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=5.59%, sys=16.59%, ctx=10788462, majf=0, minf=4503
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=7532241,3228565,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=245MiB/s (257MB/s), 245MiB/s-245MiB/s (257MB/s-257MB/s), io=28.7GiB (30.9GB), run=120001-120001msec
  WRITE: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=12.3GiB (13.2GB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=7527276/3226395, merge=0/0, ticks=708904/60504, in_queue=769408, util=100.00%, aggrios=1506448/1291658, aggrmerge=0/0, aggrticks=146758/20850, aggrin_queue=167610, aggrutil=100.00%
  nvme0n1: ios=1519277/1290683, merge=0/1, ticks=147388/18530, in_queue=165919, util=100.00%
  nvme3n1: ios=1514805/1292302, merge=0/1, ticks=147915/19343, in_queue=167258, util=100.00%
  nvme2n1: ios=1496419/1292112, merge=0/0, ticks=146194/22563, in_queue=168757, util=100.00%
  nvme1n1: ios=1507568/1291756, merge=0/1, ticks=146524/20371, in_queue=166896, util=100.00%
  nvme4n1: ios=1494172/1291439, merge=0/1, ticks=145773/23447, in_queue=169220, util=100.00%

6.5 4M 顺序写

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4m_seq_write \
  --rw=write \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=2442MiB/s][w=610 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=2086304
  write: IOPS=613, BW=2452MiB/s (2571MB/s)(144GiB/60018msec); 0 zone resets
    slat (usec): min=542, max=22108, avg=1577.38, stdev=2473.07
    clat (usec): min=3230, max=32255, avg=11468.04, stdev=5453.40
     lat (usec): min=4116, max=33108, avg=13045.85, stdev=5470.07
    clat percentiles (usec):
     |  1.00th=[ 5342],  5.00th=[ 5866], 10.00th=[ 6194], 20.00th=[ 6587],
     | 30.00th=[ 6849], 40.00th=[ 7111], 50.00th=[ 7701], 60.00th=[14746],
     | 70.00th=[16450], 80.00th=[17433], 90.00th=[18220], 95.00th=[19530],
     | 99.00th=[23725], 99.50th=[26084], 99.90th=[28443], 99.95th=[28967],
     | 99.99th=[30278]
   bw (  MiB/s): min= 2352, max= 2560, per=100.00%, avg=2453.67, stdev=40.01, samples=119
   iops        : min=  588, max=  640, avg=613.41, stdev=10.01, samples=119
  lat (msec)   : 4=0.02%, 10=54.90%, 20=41.40%, 50=3.68%
  cpu          : usr=17.28%, sys=41.36%, ctx=15137, majf=0, minf=25
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,36794,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=2452MiB/s (2571MB/s), 2452MiB/s-2452MiB/s (2571MB/s-2571MB/s), io=144GiB (154GB), run=60018-60018msec

Disk stats (read/write):
    md0: ios=0/1197325, merge=0/0, ticks=0/5861456, in_queue=5861456, util=100.00%, aggrios=0/482470, aggrmerge=0/7, aggrticks=0/1511007, aggrin_queue=1511007, aggrutil=97.70%
  nvme0n1: ios=0/482466, merge=0/7, ticks=0/1820645, in_queue=1820645, util=97.70%
  nvme3n1: ios=0/482471, merge=0/9, ticks=0/1352720, in_queue=1352719, util=96.34%
  nvme2n1: ios=0/482472, merge=0/8, ticks=0/1424437, in_queue=1424438, util=96.11%
  nvme1n1: ios=0/482473, merge=0/7, ticks=0/1480971, in_queue=1480971, util=96.62%
  nvme4n1: ios=0/482470, merge=0/7, ticks=0/1476262, in_queue=1476263, util=96.65%

6.6 4M 顺序读

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4m_seq_read \
  --rw=read \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=0): [f(1)][100.0%][r=9768MiB/s][r=2442 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=2096984
  read: IOPS=2607, BW=10.2GiB/s (10.9GB/s)(611GiB/60001msec)
    slat (usec): min=268, max=2362, avg=381.17, stdev=50.26
    clat (usec): min=416, max=14426, avg=2685.17, stdev=266.94
     lat (usec): min=810, max=16790, avg=3066.55, stdev=300.73
    clat percentiles (usec):
     |  1.00th=[ 2245],  5.00th=[ 2376], 10.00th=[ 2507], 20.00th=[ 2540],
     | 30.00th=[ 2606], 40.00th=[ 2638], 50.00th=[ 2671], 60.00th=[ 2704],
     | 70.00th=[ 2737], 80.00th=[ 2769], 90.00th=[ 2835], 95.00th=[ 2966],
     | 99.00th=[ 3982], 99.50th=[ 4228], 99.90th=[ 4752], 99.95th=[ 4948],
     | 99.99th=[ 5407]
   bw (  MiB/s): min= 9520, max=12216, per=100.00%, avg=10443.82, stdev=504.01, samples=119
   iops        : min= 2380, max= 3054, avg=2610.84, stdev=125.95, samples=119
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.09%, 4=98.94%, 10=0.96%, 20=0.01%
  cpu          : usr=0.72%, sys=99.27%, ctx=175, majf=0, minf=8214
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=156465,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=10.2GiB/s (10.9GB/s), 10.2GiB/s-10.2GiB/s (10.9GB/s-10.9GB/s), io=611GiB (656GB), run=60001-60001msec

Disk stats (read/write):
    md0: ios=5436104/2, merge=0/0, ticks=2081068/0, in_queue=2081068, util=100.00%, aggrios=1085227/15, aggrmerge=6123/0, aggrticks=411348/0, aggrin_queue=411349, aggrutil=99.90%
  nvme0n1: ios=1086197/15, merge=6120/0, ticks=410605/1, in_queue=410605, util=99.90%
  nvme3n1: ios=1085860/16, merge=6068/0, ticks=411802/0, in_queue=411803, util=99.90%
  nvme2n1: ios=1080697/17, merge=6171/0, ticks=412160/0, in_queue=412162, util=99.90%
  nvme1n1: ios=1089511/16, merge=6136/0, ticks=410432/1, in_queue=410432, util=99.90%
  nvme4n1: ios=1083870/15, merge=6120/0, ticks=411745/1, in_queue=411745, util=99.90%

7. 对比

测试结果（Latency 为 fio 报告之 lat avg，CPU 为 usr+sys）

7.1 4k - 随机写

系统	进程数	IOPS	BW	Latency	CPU
单盘	8	65.1k	254MiB/s	3932.32us	36.86%
RAID 0	8	87.9k	343MiB/s	2911.98us	50.33%
RAID 1	8	82.4k	322MiB/s	3102.50us	30.18%
RAID 5	8	29.4k	115MiB/s	8714.07us	9.93%
RAID 6	8	23.9k	93.4MiB/s	10702.73us	8.16%
RAID 10	8	61.3k	239MiB/s	4169.79us	25.71%

RAID 0 最高（87.9k IOPS / 343MiB/s），条带写多盘并行。
RAID 5/6 最低（约 23～29k），校验写带来读-改-写与校验计算开销。
RAID 1、RAID 10 介于单盘与 RAID 0 之间，镜像/镜像条带写有一定并行度。

7.2 4k - 随机读

系统	进程数	IOPS	BW	Latency	CPU
单盘	8	294k	1147MiB/s	870.84us	39.54%
RAID 0	8	1009k	3940MiB/s	253.20us	99.44%
RAID 1	8	734k	2869MiB/s	347.77us	90.69%
RAID 5	8	112k	437MiB/s	2288.70us	22.12%
RAID 6	8	999k	3902MiB/s	255.50us	97.60%
RAID 10	8	920k	3594MiB/s	277.57us	99.98%

RAID 0、RAID 6、RAID 10 接近且最高（约 920k～1009k IOPS），条带/双校验/镜像条带读均可多盘并发。
RAID 1 次之（734k），多副本可分流读。
RAID 5 明显较低（112k），单校验条带下随机读易受校验块与布局影响。

7.3 4k - 随机读写

系统	进程数	IOPS	BW	Latency	CPU
单盘	8	64.7k	256MiB/s	162.52us	10.70%
RAID 0	8	89.5k	350MiB/s	109.26us	19.52%
RAID 1	8	82.3k	321MiB/s	114.84us	18.94%
RAID 5	8	24.5k	95.6MiB/s	264.55us	6.43%
RAID 6	8	22.0k	86.3MiB/s	110.52us	5.30%
RAID 10	8	89.7k	350MiB/s	108.83us	22.18%

RAID 0、RAID 1、RAID 10 接近且较高（约 82～90k IOPS / 321～350MiB/s），读多写少时条带/镜像表现好。
RAID 5、RAID 6 较低（约 22～25k / 86～96MiB/s），混合负载下校验与写放大仍明显拉低性能。

7.4 4M - 顺序写

系统	进程数	IOPS	BW	Latency	CPU
单盘	1	237	950MiB/s	33680us	18.16%
RAID 0	1	1203	4815MiB/s	6643.65us	80.79%
RAID 1	1	242	969MiB/s	33005.52us	49.18%
RAID 5	1	233	932MiB/s	34290us	45.43%
RAID 6	1	195	780MiB/s	40950us	41.45%
RAID 10	1	613	2452MiB/s	13045.85us	58.64%

RAID 0 最高（1203 IOPS / 4815MiB/s），多盘顺序写条带化效果明显。
RAID 5、RAID 6 较低（195～233 IOPS / 780～932MiB/s），校验与重建参与顺序写。
RAID 1 接近单盘（242 / 969MiB/s），多副本写带宽受单副本限制。
RAID 10 约为 RAID 0 的一半（613 / 2452MiB/s），镜像条带写需写多份。

7.5 4M - 顺序读

系统	进程数	IOPS	BW	Latency	CPU
单盘	1	722	2890MiB/s	11065.58us	29.35%
RAID 0	1	3385	13.2GiB/s	2361.44us	97.85%
RAID 1	1	762	3049MiB/s	10491.80us	41.95%
RAID 5	1	317	1271MiB/s	25159us	38.73%
RAID 6	1	2725	10.6GiB/s	2933.25us	99.15%
RAID 10	1	2607	10.2GiB/s	3066.55us	99.99%

RAID 0、RAID 6、RAID 10 接近且最高（约 2607～3385 IOPS / 10.2～13.2GiB/s），大块顺序读多盘并发优势明显。
RAID 1 次之（762 IOPS / 3049MiB/s），多副本可分流但受单条带带宽影响。
RAID 5 明显较低（317 IOPS / 1271MiB/s），单校验条带下顺序读带宽仍受限。

8. 结论

5 盘 RAID 针对单盘的性能放大对比如下：（实测 / 理论）

RAID Level	4k 小文件随机读	4k 小文件随机写	4M 大文件顺序读	4M 大文件顺序写
RAID 0	3.4× / 4×	1.35× / 4×	4.7× / 4×	5.1× / 4×
RAID 1	2.5× / 4×	1.27× / 1×	1.1× / 4×	1.0× / 1×
RAID 5	0.38× / 3×	0.45× / 0.75×	0.44× / 3×	1.0× / 3×
RAID 6	3.4× / 2×	0.37× / 0.5×	3.8× / 2×	0.82× / 2×
RAID 10	3.1× / 4×	0.94× / 2×	3.6× / 4×	2.6× / 2×

其中，RAID 5 的数据比较差，但反复测试依然是这个结果。

此外还有 RAID 50、RAID 60，采用 RAIDx + RAID 0 的组合方式，具体选型还要结合盘的数量，如果 disk num 在 4 - 16 之间，可以采用 RAID 6、RAID 10；如果 disk num 不少于 16，可以采用 RAID 60。RAID 6、RAID 10 采用硬阵列会有明显的性能提升。