Please enable Javascript to view the contents

多机多盘 MinIO 集群在不同纠删码配置下的性能测试

 ·  ☕ 29 分钟

1. 纠删码配置

默认的纠删位配置如下:

Erasure Set SizeDefault Parity (EC:M)
1EC:0
2-3EC:1
4-5EC:2
6 - 7EC:3
8 - 16EC:4

Parity 最大值是 ERASURE_SET_SIZE/2，也就是校验位不能超过数据位。

参考 https://docs.min.io/enterprise/aistor-object-store/reference/aistor-server/settings/storage-class/

2. minio 集群环境

2.1 创建 minio 集群

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

export CONTAINER_CLI=nerdctl
export IMAGE=registry.cn-beijing.aliyuncs.com/opshub/minio-minio:RELEASE.2025-04-22T22-12-26Z

export ROOT_USER=minioadmin
export ROOT_PASSWORD=minioadmin

export MINIO_ERASURE_SET_DRIVE_COUNT=16
export MINIO_STORAGE_CLASS_STANDARD=EC:6

export POOL_0="http://minio{1...4}/mnt/data{0...3}"

$CONTAINER_CLI run -d \
  --net host \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  --ulimit nofile=1048576:1048576 \
  --memory-swappiness=0 \
  --name minio \
  -v /mnt/data0:/mnt/data0 \
  -v /mnt/data1:/mnt/data1 \
  -v /mnt/data2:/mnt/data2 \
  -v /mnt/data3:/mnt/data3 \
  -e "MINIO_ROOT_USER=$ROOT_USER" \
  -e "MINIO_ROOT_PASSWORD=$ROOT_PASSWORD" \
  -e "MINIO_ERASURE_SET_DRIVE_COUNT=$MINIO_ERASURE_SET_DRIVE_COUNT" \
  -e "MINIO_STORAGE_CLASS_STANDARD=$MINIO_STORAGE_CLASS_STANDARD" \
  $IMAGE server \
  $POOL_0 \
  --console-address ":9090" \
  --address ":9000"

2.2 清理 minio 集群

1
2
3
4
5

$CONTAINER_CLI rm -f minio
for i in {0..3}; do
  rm -rf /mnt/data${i}/*
  rm -rf /mnt/data${i}/.minio.sys
done

3. 2个数据位2个校验位

3.1 集群拓扑

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

●  minio1:9000
   Uptime: 5 minutes
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

●  minio2:9000
   Uptime: 5 minutes
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

●  minio3:9000
   Uptime: 5 minutes
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

●  minio4:9000
   Uptime: 5 minutes
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

┌──────┬───────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage          │ Erasure stripe size │ Erasure sets │
│ 1st  │ 0.7% (total: 5.8 TiB) │ 4                   │ 4            │
└──────┴───────────────────────┴─────────────────────┴──────────────┘

16 drives online, 0 drives offline, EC:2

3.2 4KiB 文件

  • get
1
2
3
4
5
6
7
8

warp get \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4KiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

Reqs: 4157472, Errs:0, Objs:4157472, Bytes: 15.86GiB
 -       GET Average: 13896 Obj/s, 54.3MiB/s; Current 14617 Obj/s, 57.1MiB/s, 2.2 ms/req, TTFB: 2.2ms


Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 54.29 MiB/s, 13897.18 obj/s
 * Reqs: Avg: 2.2ms, 50%: 2.0ms, 90%: 2.7ms, 99%: 6.3ms, Fastest: 1.3ms, Slowest: 212.4ms, StdDev: 1.1ms
 * TTFB: Avg: 2ms, Best: 1ms, 25th: 2ms, Median: 2ms, 75th: 2ms, 90th: 3ms, 99th: 6ms, Worst: 212ms StdDev: 1ms

Throughput by host:
 * http://minio1:9000: Avg: 14.84 MiB/s, 3799.16 obj/s
 * http://minio2:9000: Avg: 13.35 MiB/s, 3418.40 obj/s
 * http://minio3:9000: Avg: 13.06 MiB/s, 3343.50 obj/s
 * http://minio4:9000: Avg: 12.95 MiB/s, 3314.76 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 60.2MiB/s, 15408.76 obj/s
 * 50% Median: 56.6MiB/s, 14488.53 obj/s
 * Slowest: 27.5MiB/s, 7046.40 obj/s
  • put
1
2
3
4
5
6
7
8

warp put \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4KiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

Reqs: 2803801, Errs:0, Objs:2803801, Bytes: 10.70GiB
 -       PUT Average: 9373 Obj/s, 36.6MiB/s; Current 9407 Obj/s, 36.7MiB/s, 3.4 ms/req


Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 36.61 MiB/s, 9372.83 obj/s
 * Reqs: Avg: 3.4ms, 50%: 3.0ms, 90%: 4.4ms, 99%: 10.3ms, Fastest: 2.0ms, Slowest: 213.9ms, StdDev: 2.0ms

Throughput by host:
 * http://minio1:9000: Avg: 9.68 MiB/s, 2478.44 obj/s
 * http://minio2:9000: Avg: 9.00 MiB/s, 2302.74 obj/s
 * http://minio3:9000: Avg: 8.97 MiB/s, 2295.51 obj/s
 * http://minio4:9000: Avg: 8.95 MiB/s, 2292.45 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 39.8MiB/s, 10191.18 obj/s
 * 50% Median: 36.7MiB/s, 9407.68 obj/s
 * Slowest: 31.8MiB/s, 8138.45 obj/s
  • mixed
1
2
3
4
5
6
7
8

warp mixed \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4KiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

Reqs: 3544989, Errs:0, Objs:3544989, Bytes: 8308.6MiB
 -    DELETE Average: 1185 Obj/s; Current 1171 Obj/s, 4.2 ms/req
 -       GET Average: 5334 Obj/s, 20.8MiB/s; Current 5301 Obj/s, 20.7MiB/s, 2.2 ms/req, TTFB: 2.2ms
 -       PUT Average: 1778 Obj/s, 6.9MiB/s; Current 1758 Obj/s, 6.9MiB/s, 4.0 ms/req
 -      STAT Average: 3556 Obj/s; Current 3527 Obj/s, 1.8 ms/req


Report: DELETE. Concurrency: 32. Ran: 4m57s
 * Average: 1185.54 obj/s
 * Reqs: Avg: 4.4ms, 50%: 3.3ms, 90%: 5.3ms, 99%: 29.5ms, Fastest: 2.4ms, Slowest: 226.5ms, StdDev: 4.9ms

Throughput by host:
 * http://minio1:9000: Avg: 322.96 obj/s
 * http://minio2:9000: Avg: 288.11 obj/s
 * http://minio3:9000: Avg: 287.50 obj/s
 * http://minio4:9000: Avg: 284.84 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 1358.65 obj/s
 * 50% Median: 1227.79 obj/s
 * Slowest: 612.39 obj/s

──────────────────────────────────

Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 20.84 MiB/s, 5334.88 obj/s
 * Reqs: Avg: 2.2ms, 50%: 2.0ms, 90%: 2.7ms, 99%: 6.5ms, Fastest: 1.4ms, Slowest: 210.0ms, StdDev: 1.1ms
 * TTFB: Avg: 2ms, Best: 1ms, 25th: 2ms, Median: 2ms, 75th: 2ms, 90th: 3ms, 99th: 6ms, Worst: 210ms StdDev: 1ms

Throughput by host:
 * http://minio1:9000: Avg: 5.54 MiB/s, 1417.50 obj/s
 * http://minio2:9000: Avg: 5.12 MiB/s, 1310.17 obj/s
 * http://minio3:9000: Avg: 5.07 MiB/s, 1298.39 obj/s
 * http://minio4:9000: Avg: 5.08 MiB/s, 1299.22 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 23.9MiB/s, 6110.13 obj/s
 * 50% Median: 21.6MiB/s, 5521.49 obj/s
 * Slowest: 10.8MiB/s, 2754.17 obj/s

──────────────────────────────────

Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 6.95 MiB/s, 1778.32 obj/s
 * Reqs: Avg: 4.2ms, 50%: 3.1ms, 90%: 6.0ms, 99%: 22.5ms, Fastest: 2.1ms, Slowest: 418.2ms, StdDev: 4.0ms

Throughput by host:
 * http://minio1:9000: Avg: 1.82 MiB/s, 466.39 obj/s
 * http://minio2:9000: Avg: 1.70 MiB/s, 435.50 obj/s
 * http://minio3:9000: Avg: 1.71 MiB/s, 437.69 obj/s
 * http://minio4:9000: Avg: 1.70 MiB/s, 435.55 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 8.0MiB/s, 2042.46 obj/s
 * 50% Median: 7.2MiB/s, 1838.96 obj/s
 * Slowest: 3.6MiB/s, 921.19 obj/s

──────────────────────────────────

Report: STAT. Concurrency: 32. Ran: 4m57s
 * Average: 3556.59 obj/s
 * Reqs: Avg: 1.9ms, 50%: 1.7ms, 90%: 2.2ms, 99%: 6.2ms, Fastest: 1.1ms, Slowest: 37.8ms, StdDev: 1.1ms

Throughput by host:
 * http://minio1:9000: Avg: 967.55 obj/s
 * http://minio2:9000: Avg: 870.76 obj/s
 * http://minio3:9000: Avg: 855.63 obj/s
 * http://minio4:9000: Avg: 856.26 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 4075.70 obj/s
 * 50% Median: 3679.04 obj/s
 * Slowest: 1836.48 obj/s


──────────────────────────────────

Report: Total. Concurrency: 32. Ran: 4m57s
 * Average: 27.79 MiB/s, 11855.33 obj/s

Throughput by host:
 * http://minio1:9000: Avg: 7.36 MiB/s, 3174.39 obj/s
 * http://minio2:9000: Avg: 6.82 MiB/s, 2904.51 obj/s
 * http://minio3:9000: Avg: 6.78 MiB/s, 2879.19 obj/s
 * http://minio4:9000: Avg: 6.78 MiB/s, 2875.85 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 31.8MiB/s, 13586.94 obj/s
 * 50% Median: 28.7MiB/s, 12267.08 obj/s
 * Slowest: 14.4MiB/s, 6124.23 obj/s

3.3 4MiB 文件

  • get
1
2
3
4
5
6
7
8

warp get \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

Reqs: 91868, Errs:0, Objs:91868, Bytes: 358.86GiB
 -       GET Average: 307 Obj/s, 1228.9MiB/s; Current 299 Obj/s, 1195.3MiB/s, 104.7 ms/req, TTFB: 30.3m


Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 1228.90 MiB/s, 307.23 obj/s
 * Reqs: Avg: 104.0ms, 50%: 97.0ms, 90%: 148.9ms, 99%: 215.8ms, Fastest: 21.5ms, Slowest: 874.2ms, StdDev: 38.7ms
 * TTFB: Avg: 30ms, Best: 9ms, 25th: 23ms, Median: 28ms, 75th: 36ms, 90th: 41ms, 99th: 49ms, Worst: 488ms StdDev: 14ms

Throughput by host:
 * http://minio1:9000: Avg: 427.19 MiB/s, 106.80 obj/s
 * http://minio2:9000: Avg: 309.71 MiB/s, 77.43 obj/s
 * http://minio3:9000: Avg: 245.53 MiB/s, 61.38 obj/s
 * http://minio4:9000: Avg: 246.49 MiB/s, 61.62 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 1285.3MiB/s, 321.33 obj/s
 * 50% Median: 1229.6MiB/s, 307.41 obj/s
 * Slowest: 1156.9MiB/s, 289.24 obj/s
  • put
1
2
3
4
5
6
7
8

warp put \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

Reqs: 74746, Errs:0, Objs:74746, Bytes: 291.98GiB
 -       PUT Average: 250 Obj/s, 998.3MiB/s; Current 256 Obj/s, 1022.2MiB/s, 128.3 ms/req


Report: PUT. Concurrency: 32. Ran: 4m58s
 * Average: 998.27 MiB/s, 249.57 obj/s
 * Reqs: Avg: 127.9ms, 50%: 107.7ms, 90%: 194.6ms, 99%: 354.3ms, Fastest: 47.9ms, Slowest: 1100.1ms, StdDev: 58.8ms

Throughput by host:
 * http://minio1:9000: Avg: 233.20 MiB/s, 58.30 obj/s
 * http://minio2:9000: Avg: 253.72 MiB/s, 63.43 obj/s
 * http://minio3:9000: Avg: 272.13 MiB/s, 68.03 obj/s
 * http://minio4:9000: Avg: 239.09 MiB/s, 59.77 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 1058.6MiB/s, 264.64 obj/s
 * 50% Median: 998.3MiB/s, 249.59 obj/s
 * Slowest: 956.6MiB/s, 239.14 obj/s
  • mixed
1
2
3
4
5
6
7
8

warp mixed \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

Reqs: 178486, Errs:0, Objs:178486, Bytes: 418.31GiB
 -    DELETE Average: 60 Obj/s; Current 51 Obj/s, 25.2 ms/req
 -       GET Average: 269 Obj/s, 1075.3MiB/s; Current 270 Obj/s, 1081.7MiB/s, 68.3 ms/req, TTFB: 23.9ms
 -       PUT Average: 90 Obj/s, 358.6MiB/s; Current 88 Obj/s, 350.7MiB/s, 104.2 ms/req
 -      STAT Average: 179 Obj/s; Current 183 Obj/s, 14.0 ms/req


Report: DELETE. Concurrency: 32. Ran: 4m57s
 * Average: 59.73 obj/s
 * Reqs: Avg: 25.2ms, 50%: 24.8ms, 90%: 32.1ms, 99%: 41.6ms, Fastest: 7.9ms, Slowest: 258.5ms, StdDev: 6.6ms

Throughput by host:
 * http://minio1:9000: Avg: 20.57 obj/s
 * http://minio2:9000: Avg: 18.12 obj/s
 * http://minio3:9000: Avg: 10.43 obj/s
 * http://minio4:9000: Avg: 10.65 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 72.65 obj/s
 * 50% Median: 60.22 obj/s
 * Slowest: 47.06 obj/s

──────────────────────────────────

Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 1075.37 MiB/s, 268.84 obj/s
 * Reqs: Avg: 68.5ms, 50%: 61.3ms, 90%: 107.6ms, 99%: 144.1ms, Fastest: 16.5ms, Slowest: 474.9ms, StdDev: 31.4ms
 * TTFB: Avg: 24ms, Best: 6ms, 25th: 19ms, Median: 22ms, 75th: 27ms, 90th: 33ms, 99th: 42ms, Worst: 305ms StdDev: 12ms

Throughput by host:
 * http://minio1:9000: Avg: 358.67 MiB/s, 89.67 obj/s
 * http://minio2:9000: Avg: 324.29 MiB/s, 81.07 obj/s
 * http://minio3:9000: Avg: 195.88 MiB/s, 48.97 obj/s
 * http://minio4:9000: Avg: 196.49 MiB/s, 49.12 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 1164.3MiB/s, 291.07 obj/s
 * 50% Median: 1076.4MiB/s, 269.11 obj/s
 * Slowest: 1004.9MiB/s, 251.23 obj/s

──────────────────────────────────

Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 358.46 MiB/s, 89.61 obj/s
 * Reqs: Avg: 105.2ms, 50%: 85.6ms, 90%: 164.0ms, 99%: 186.9ms, Fastest: 44.4ms, Slowest: 427.2ms, StdDev: 42.4ms

Throughput by host:
 * http://minio1:9000: Avg: 117.28 MiB/s, 29.32 obj/s
 * http://minio2:9000: Avg: 109.33 MiB/s, 27.33 obj/s
 * http://minio3:9000: Avg: 66.56 MiB/s, 16.64 obj/s
 * http://minio4:9000: Avg: 65.41 MiB/s, 16.35 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 402.9MiB/s, 100.73 obj/s
 * 50% Median: 360.2MiB/s, 90.05 obj/s
 * Slowest: 308.7MiB/s, 77.17 obj/s

──────────────────────────────────

Report: STAT. Concurrency: 32. Ran: 4m57s
 * Average: 179.19 obj/s
 * Reqs: Avg: 14.3ms, 50%: 13.9ms, 90%: 19.1ms, 99%: 23.6ms, Fastest: 2.5ms, Slowest: 237.7ms, StdDev: 6.1ms

Throughput by host:
 * http://minio1:9000: Avg: 60.35 obj/s
 * http://minio2:9000: Avg: 53.04 obj/s
 * http://minio3:9000: Avg: 32.76 obj/s
 * http://minio4:9000: Avg: 33.12 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 199.85 obj/s
 * 50% Median: 179.00 obj/s
 * Slowest: 159.35 obj/s


──────────────────────────────────

Report: Total. Concurrency: 32. Ran: 4m57s
 * Average: 1433.83 MiB/s, 597.38 obj/s

Throughput by host:
 * http://minio1:9000: Avg: 475.94 MiB/s, 199.89 obj/s
 * http://minio2:9000: Avg: 433.61 MiB/s, 179.55 obj/s
 * http://minio3:9000: Avg: 262.43 MiB/s, 108.79 obj/s
 * http://minio4:9000: Avg: 261.90 MiB/s, 109.24 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 1494.9MiB/s, 636.73 obj/s
 * 50% Median: 1447.3MiB/s, 598.05 obj/s
 * Slowest: 1349.0MiB/s, 552.81 obj/s

3.4 100MiB 文件

  • get
1
2
3
4
5
6
7
8

warp get \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 100MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

Reqs: 3727, Errs:0, Objs:3727, Bytes: 363.96GiB
 -       GET Average: 12 Obj/s, 1244.8MiB/s; Current 8 Obj/s, 819.5MiB/s, 2579.5 ms/req, TTFB: 36.2ms


Report: GET. Concurrency: 32. Ran: 4m59s
 * Average: 1246.97 MiB/s, 12.47 obj/s
 * Reqs: Avg: 2557.8ms, 50%: 2707.2ms, 90%: 3576.9ms, 99%: 4061.4ms, Fastest: 370.3ms, Slowest: 4476.4ms, StdDev: 814.2ms
 * TTFB: Avg: 36ms, Best: 8ms, 25th: 26ms, Median: 34ms, 75th: 43ms, 90th: 46ms, 99th: 146ms, Worst: 263ms StdDev: 16ms

Throughput by host:
 * http://minio1:9000: Avg: 480.07 MiB/s, 4.80 obj/s
 * http://minio2:9000: Avg: 272.45 MiB/s, 2.72 obj/s
 * http://minio3:9000: Avg: 248.12 MiB/s, 2.48 obj/s
 * http://minio4:9000: Avg: 246.11 MiB/s, 2.46 obj/s

Throughput, split into 299 x 1s:
 * Fastest: 1406.1MiB/s, 14.06 obj/s
 * 50% Median: 1243.0MiB/s, 12.43 obj/s
 * Slowest: 1135.1MiB/s, 11.35 obj/s
  • put
1
2
3
4
5
6
7
8

warp put \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 100MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

Reqs: 3085, Errs:0, Objs:3085, Bytes: 301.27GiB
 -       PUT Average: 10 Obj/s, 1031.8MiB/s; Current 10 Obj/s, 966.7MiB/s, 3104.4 ms/req


Report: PUT. Concurrency: 32. Ran: 4m59s
 * Average: 1034.00 MiB/s, 10.34 obj/s
 * Reqs: Avg: 3093.8ms, 50%: 2911.7ms, 90%: 4766.4ms, 99%: 6661.2ms, Fastest: 1037.0ms, Slowest: 7682.2ms, StdDev: 1157.5ms

Throughput by host:
 * http://minio1:9000: Avg: 218.68 MiB/s, 2.19 obj/s
 * http://minio2:9000: Avg: 261.46 MiB/s, 2.61 obj/s
 * http://minio3:9000: Avg: 277.05 MiB/s, 2.77 obj/s
 * http://minio4:9000: Avg: 276.35 MiB/s, 2.76 obj/s

Throughput, split into 299 x 1s:
 * Fastest: 1335.0MiB/s, 13.35 obj/s
 * 50% Median: 1033.2MiB/s, 10.33 obj/s
 * Slowest: 902.9MiB/s, 9.03 obj/s
  • mixed
1
2
3
4
5
6
7
8

warp mixed \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 100MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

Reqs: 7512, Errs:0, Objs:7512, Bytes: 440.33GiB
 -    DELETE Average: 2 Obj/s; Current 1 Obj/s, 34.7 ms/req
 -       GET Average: 11 Obj/s, 1125.7MiB/s; Current 8 Obj/s, 821.4MiB/s, 2474.7 ms/req, TTFB: 39.0ms
 -       PUT Average: 4 Obj/s, 375.1MiB/s; Current 6 Obj/s, 564.6MiB/s, 1122.1 ms/req
 -      STAT Average: 7 Obj/s; Current 3 Obj/s, 18.8 ms/req


Report: DELETE. Concurrency: 32. Ran: 4m56s
 * Average: 2.50 obj/s
 * Reqs: Avg: 35.0ms, 50%: 34.0ms, 90%: 41.4ms, 99%: 72.2ms, Fastest: 5.9ms, Slowest: 246.8ms, StdDev: 9.8ms

Throughput by host:
 * http://minio1:9000: Avg: 0.99 obj/s
 * http://minio2:9000: Avg: 0.62 obj/s
 * http://minio3:9000: Avg: 0.49 obj/s
 * http://minio4:9000: Avg: 0.45 obj/s

Throughput, split into 296 x 1s:
 * Fastest: 7.45 obj/s
 * 50% Median: 2.00 obj/s
 * Slowest: 0.00 obj/s

──────────────────────────────────

Report: GET. Concurrency: 32. Ran: 4m58s
 * Average: 1126.63 MiB/s, 11.27 obj/s
 * Reqs: Avg: 2447.5ms, 50%: 2649.6ms, 90%: 3563.1ms, 99%: 4039.0ms, Fastest: 230.3ms, Slowest: 4605.5ms, StdDev: 921.2ms
 * TTFB: Avg: 37ms, Best: 7ms, 25th: 27ms, Median: 35ms, 75th: 43ms, 90th: 48ms, 99th: 208ms, Worst: 1.028s StdDev: 22ms

Throughput by host:
 * http://minio1:9000: Avg: 452.33 MiB/s, 4.52 obj/s
 * http://minio2:9000: Avg: 242.91 MiB/s, 2.43 obj/s
 * http://minio3:9000: Avg: 215.91 MiB/s, 2.16 obj/s
 * http://minio4:9000: Avg: 214.93 MiB/s, 2.15 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 1362.0MiB/s, 13.62 obj/s
 * 50% Median: 1125.3MiB/s, 11.25 obj/s
 * Slowest: 819.9MiB/s, 8.20 obj/s

──────────────────────────────────

Report: PUT. Concurrency: 32. Ran: 4m58s
 * Average: 375.12 MiB/s, 3.75 obj/s
 * Reqs: Avg: 1095.6ms, 50%: 969.5ms, 90%: 1552.8ms, 99%: 1702.5ms, Fastest: 535.4ms, Slowest: 2021.1ms, StdDev: 341.5ms

Throughput by host:
 * http://minio1:9000: Avg: 148.83 MiB/s, 1.49 obj/s
 * http://minio2:9000: Avg: 90.42 MiB/s, 0.90 obj/s
 * http://minio3:9000: Avg: 68.40 MiB/s, 0.68 obj/s
 * http://minio4:9000: Avg: 69.09 MiB/s, 0.69 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 896.7MiB/s, 8.97 obj/s
 * 50% Median: 382.4MiB/s, 3.82 obj/s
 * Slowest: 0.00 obj/s

──────────────────────────────────

Report: STAT. Concurrency: 32. Ran: 4m57s
 * Average: 7.46 obj/s
 * Reqs: Avg: 19.6ms, 50%: 19.8ms, 90%: 23.5ms, 99%: 72.8ms, Fastest: 1.9ms, Slowest: 233.1ms, StdDev: 8.3ms

Throughput by host:
 * http://minio1:9000: Avg: 2.92 obj/s
 * http://minio2:9000: Avg: 1.72 obj/s
 * http://minio3:9000: Avg: 1.40 obj/s
 * http://minio4:9000: Avg: 1.49 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 17.82 obj/s
 * 50% Median: 7.00 obj/s
 * Slowest: 0.00 obj/s


──────────────────────────────────

Report: Total. Concurrency: 32. Ran: 4m58s
 * Average: 1501.76 MiB/s, 24.96 obj/s

Throughput by host:
 * http://minio1:9000: Avg: 600.45 MiB/s, 9.89 obj/s
 * http://minio2:9000: Avg: 332.83 MiB/s, 5.63 obj/s
 * http://minio3:9000: Avg: 283.57 MiB/s, 4.69 obj/s
 * http://minio4:9000: Avg: 283.92 MiB/s, 4.75 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 1667.3MiB/s, 39.67 obj/s
 * 50% Median: 1571.6MiB/s, 24.51 obj/s
 * Slowest: 1461.0MiB/s, 15.61 obj/s

4. 12个数据位4个校验位

4.1 集群拓扑

这是 minio 针对四节点，每个节点四盘位置默认配置的集群拓扑。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

●  minio1:9000
   Uptime: 18 seconds
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

●  minio2:9000
   Uptime: 15 seconds
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

●  minio3:9000
   Uptime: 21 seconds
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

●  minio4:9000
   Uptime: 13 seconds
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

┌──────┬───────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage          │ Erasure stripe size │ Erasure sets │
│ 1st  │ 0.4% (total: 8.7 TiB) │ 16                  │ 1            │
└──────┴───────────────────────┴─────────────────────┴──────────────┘

16 drives online, 0 drives offline, EC:4

4.2 4KiB 文件

  • get
1
2
3
4
5
6
7
8

warp get \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4KiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

Reqs: 2922322, Errs:0, Objs:2922322, Bytes: 11.15GiB
 -       GET Average: 9769 Obj/s, 38.2MiB/s; Current 10178 Obj/s, 39.8MiB/s, 3.2 ms/req, TTFB: 3.2ms


Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 38.16 MiB/s, 9769.07 obj/s
 * Reqs: Avg: 3.2ms, 50%: 2.6ms, 90%: 5.0ms, 99%: 11.1ms, Fastest: 1.6ms, Slowest: 217.9ms, StdDev: 2.0ms
 * TTFB: Avg: 3ms, Best: 2ms, 25th: 2ms, Median: 3ms, 75th: 3ms, 90th: 5ms, 99th: 11ms, Worst: 218ms StdDev: 2ms

Throughput by host:
 * http://minio1:9000: Avg: 10.15 MiB/s, 2599.33 obj/s
 * http://minio2:9000: Avg: 9.50 MiB/s, 2430.86 obj/s
 * http://minio3:9000: Avg: 9.14 MiB/s, 2341.10 obj/s
 * http://minio4:9000: Avg: 9.33 MiB/s, 2388.09 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 42.6MiB/s, 10906.14 obj/s
 * 50% Median: 39.0MiB/s, 9988.74 obj/s
 * Slowest: 27.0MiB/s, 6900.40 obj/s
  • put
1
2
3
4
5
6
7
8

warp put \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4KiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

Reqs: 1376871, Errs:0, Objs:1376871, Bytes: 5378.4MiB
 -       PUT Average: 4596 Obj/s, 18.0MiB/s; Current 4631 Obj/s, 18.1MiB/s, 7.1 ms/req


Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 17.95 MiB/s, 4595.68 obj/s
 * Reqs: Avg: 7.0ms, 50%: 6.0ms, 90%: 9.8ms, 99%: 20.9ms, Fastest: 2.5ms, Slowest: 220.4ms, StdDev: 3.4ms

Throughput by host:
 * http://minio1:9000: Avg: 4.55 MiB/s, 1166.03 obj/s
 * http://minio2:9000: Avg: 4.43 MiB/s, 1134.47 obj/s
 * http://minio3:9000: Avg: 4.47 MiB/s, 1144.45 obj/s
 * http://minio4:9000: Avg: 4.50 MiB/s, 1151.13 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 19.7MiB/s, 5030.44 obj/s
 * 50% Median: 18.0MiB/s, 4598.12 obj/s
 * Slowest: 15.6MiB/s, 3994.15 obj/s
  • mixed
1
2
3
4
5
6
7
8

warp mixed \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4KiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

Reqs: 2197067, Errs:0, Objs:2197067, Bytes: 5149.4MiB
 -    DELETE Average: 734 Obj/s; Current 665 Obj/s, 9.0 ms/req
 -       GET Average: 3304 Obj/s, 12.9MiB/s; Current 2972 Obj/s, 11.6MiB/s, 3.0 ms/req, TTFB: 3.0ms
 -       PUT Average: 1101 Obj/s, 4.3MiB/s; Current 988 Obj/s, 3.9MiB/s, 8.7 ms/req
 -      STAT Average: 2203 Obj/s; Current 1975 Obj/s, 2.6 ms/req


Report: DELETE. Concurrency: 32. Ran: 4m57s
 * Average: 734.09 obj/s
 * Reqs: Avg: 9.0ms, 50%: 5.6ms, 90%: 16.8ms, 99%: 52.8ms, Fastest: 3.4ms, Slowest: 210.9ms, StdDev: 9.5ms

Throughput by host:
 * http://minio1:9000: Avg: 191.46 obj/s
 * http://minio2:9000: Avg: 182.37 obj/s
 * http://minio3:9000: Avg: 178.87 obj/s
 * http://minio4:9000: Avg: 181.01 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 895.07 obj/s
 * 50% Median: 736.10 obj/s
 * Slowest: 559.14 obj/s

──────────────────────────────────

Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 12.90 MiB/s, 3303.34 obj/s
 * Reqs: Avg: 3.0ms, 50%: 2.5ms, 90%: 4.5ms, 99%: 10.5ms, Fastest: 1.6ms, Slowest: 217.9ms, StdDev: 1.8ms
 * TTFB: Avg: 3ms, Best: 2ms, 25th: 2ms, Median: 2ms, 75th: 3ms, 90th: 4ms, 99th: 10ms, Worst: 218ms StdDev: 2ms

Throughput by host:
 * http://minio1:9000: Avg: 3.35 MiB/s, 858.84 obj/s
 * http://minio2:9000: Avg: 3.20 MiB/s, 819.02 obj/s
 * http://minio3:9000: Avg: 3.15 MiB/s, 805.92 obj/s
 * http://minio4:9000: Avg: 3.19 MiB/s, 817.88 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 15.7MiB/s, 4030.54 obj/s
 * 50% Median: 13.0MiB/s, 3319.34 obj/s
 * Slowest: 9.9MiB/s, 2535.04 obj/s

──────────────────────────────────

Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 4.30 MiB/s, 1101.15 obj/s
 * Reqs: Avg: 8.4ms, 50%: 5.0ms, 90%: 16.9ms, 99%: 50.1ms, Fastest: 2.7ms, Slowest: 138.3ms, StdDev: 9.1ms

Throughput by host:
 * http://minio1:9000: Avg: 1.11 MiB/s, 284.97 obj/s
 * http://minio2:9000: Avg: 1.07 MiB/s, 272.66 obj/s
 * http://minio3:9000: Avg: 1.06 MiB/s, 270.13 obj/s
 * http://minio4:9000: Avg: 1.07 MiB/s, 272.81 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 5.3MiB/s, 1344.89 obj/s
 * 50% Median: 4.3MiB/s, 1105.95 obj/s
 * Slowest: 3.3MiB/s, 851.91 obj/s

──────────────────────────────────

Report: STAT. Concurrency: 32. Ran: 4m57s
 * Average: 2202.21 obj/s
 * Reqs: Avg: 2.6ms, 50%: 2.1ms, 90%: 3.9ms, 99%: 10.1ms, Fastest: 1.3ms, Slowest: 70.3ms, StdDev: 1.7ms

Throughput by host:
 * http://minio1:9000: Avg: 572.93 obj/s
 * http://minio2:9000: Avg: 549.39 obj/s
 * http://minio3:9000: Avg: 534.76 obj/s
 * http://minio4:9000: Avg: 544.05 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 2688.45 obj/s
 * 50% Median: 2214.78 obj/s
 * Slowest: 1690.54 obj/s


──────────────────────────────────

Report: Total. Concurrency: 32. Ran: 4m57s
 * Average: 17.21 MiB/s, 7340.80 obj/s

Throughput by host:
 * http://minio1:9000: Avg: 4.47 MiB/s, 1908.16 obj/s
 * http://minio2:9000: Avg: 4.26 MiB/s, 1823.40 obj/s
 * http://minio3:9000: Avg: 4.20 MiB/s, 1789.67 obj/s
 * http://minio4:9000: Avg: 4.26 MiB/s, 1815.70 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 21.0MiB/s, 8958.95 obj/s
 * 50% Median: 17.2MiB/s, 7370.68 obj/s
 * Slowest: 13.2MiB/s, 5636.64 obj/s

4.3 4MiB 文件

  • get
1
2
3
4
5
6
7
8

warp get \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

Reqs: 94332, Errs:0, Objs:94332, Bytes: 368.48GiB
 -       GET Average: 315 Obj/s, 1261.9MiB/s; Current 313 Obj/s, 1250.7MiB/s, 102.0 ms/req, TTFB: 31.5m


Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 1261.87 MiB/s, 315.47 obj/s
 * Reqs: Avg: 101.6ms, 50%: 97.0ms, 90%: 166.9ms, 99%: 326.6ms, Fastest: 21.6ms, Slowest: 803.8ms, StdDev: 60.0ms
 * TTFB: Avg: 31ms, Best: 10ms, 25th: 24ms, Median: 27ms, 75th: 29ms, 90th: 34ms, 99th: 238ms, Worst: 485ms StdDev: 30ms

Throughput by host:
 * http://minio1:9000: Avg: 547.15 MiB/s, 136.79 obj/s
 * http://minio2:9000: Avg: 243.65 MiB/s, 60.91 obj/s
 * http://minio3:9000: Avg: 236.05 MiB/s, 59.01 obj/s
 * http://minio4:9000: Avg: 234.60 MiB/s, 58.65 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 1314.8MiB/s, 328.69 obj/s
 * 50% Median: 1263.3MiB/s, 315.83 obj/s
 * Slowest: 1188.7MiB/s, 297.17 obj/s
  • put
1
2
3
4
5
6
7
8

warp put \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

Report: PUT. Concurrency: 32. Ran: 4m58s
 * Average: 1117.66 MiB/s, 279.41 obj/s
 * Reqs: Avg: 114.5ms, 50%: 113.6ms, 90%: 173.4ms, 99%: 292.6ms, Fastest: 38.5ms, Slowest: 657.4ms, StdDev: 51.2ms

Throughput by host:
 * http://minio1:9000: Avg: 458.76 MiB/s, 114.69 obj/s
 * http://minio2:9000: Avg: 220.59 MiB/s, 55.15 obj/s
 * http://minio3:9000: Avg: 220.66 MiB/s, 55.16 obj/s
 * http://minio4:9000: Avg: 217.58 MiB/s, 54.40 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 1143.4MiB/s, 285.85 obj/s
 * 50% Median: 1117.4MiB/s, 279.35 obj/s
 * Slowest: 1076.5MiB/s, 269.12 obj/s
  • mixed
1
2
3
4
5
6
7
8

warp mixed \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

Reqs: 189330, Errs:0, Objs:189330, Bytes: 443.70GiB
 -    DELETE Average: 63 Obj/s; Current 73 Obj/s, 26.5 ms/req
 -       GET Average: 284 Obj/s, 1136.5MiB/s; Current 281 Obj/s, 1123.1MiB/s, 64.0 ms/req, TTFB: 23.3ms
 -       PUT Average: 95 Obj/s, 378.8MiB/s; Current 98 Obj/s, 391.0MiB/s, 98.3 ms/req
 -      STAT Average: 189 Obj/s; Current 185 Obj/s, 14.9 ms/req


Report: DELETE. Concurrency: 32. Ran: 4m57s
 * Average: 63.16 obj/s
 * Reqs: Avg: 28.0ms, 50%: 25.9ms, 90%: 37.7ms, 99%: 70.6ms, Fastest: 8.4ms, Slowest: 266.0ms, StdDev: 12.4ms

Throughput by host:
 * http://minio1:9000: Avg: 24.42 obj/s
 * http://minio2:9000: Avg: 15.59 obj/s
 * http://minio3:9000: Avg: 11.64 obj/s
 * http://minio4:9000: Avg: 11.51 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 75.05 obj/s
 * 50% Median: 63.24 obj/s
 * Slowest: 50.00 obj/s

──────────────────────────────────

Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 1136.46 MiB/s, 284.12 obj/s
 * Reqs: Avg: 64.8ms, 50%: 63.5ms, 90%: 93.9ms, 99%: 273.7ms, Fastest: 10.9ms, Slowest: 1046.2ms, StdDev: 39.4ms
 * TTFB: Avg: 24ms, Best: 6ms, 25th: 18ms, Median: 21ms, 75th: 24ms, 90th: 28ms, 99th: 158ms, Worst: 1.029s StdDev: 22ms

Throughput by host:
 * http://minio1:9000: Avg: 430.10 MiB/s, 107.53 obj/s
 * http://minio2:9000: Avg: 283.08 MiB/s, 70.77 obj/s
 * http://minio3:9000: Avg: 214.93 MiB/s, 53.73 obj/s
 * http://minio4:9000: Avg: 208.37 MiB/s, 52.09 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 1199.0MiB/s, 299.75 obj/s
 * 50% Median: 1136.7MiB/s, 284.16 obj/s
 * Slowest: 1064.7MiB/s, 266.17 obj/s

──────────────────────────────────

Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 378.81 MiB/s, 94.70 obj/s
 * Reqs: Avg: 96.0ms, 50%: 69.3ms, 90%: 157.0ms, 99%: 279.3ms, Fastest: 36.9ms, Slowest: 1048.5ms, StdDev: 49.7ms

Throughput by host:
 * http://minio1:9000: Avg: 141.88 MiB/s, 35.47 obj/s
 * http://minio2:9000: Avg: 95.25 MiB/s, 23.81 obj/s
 * http://minio3:9000: Avg: 69.80 MiB/s, 17.45 obj/s
 * http://minio4:9000: Avg: 72.02 MiB/s, 18.01 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 422.3MiB/s, 105.57 obj/s
 * 50% Median: 379.5MiB/s, 94.89 obj/s
 * Slowest: 334.0MiB/s, 83.50 obj/s

──────────────────────────────────

Report: STAT. Concurrency: 32. Ran: 4m57s
 * Average: 189.39 obj/s
 * Reqs: Avg: 14.4ms, 50%: 13.7ms, 90%: 19.4ms, 99%: 33.7ms, Fastest: 2.5ms, Slowest: 236.4ms, StdDev: 8.4ms

Throughput by host:
 * http://minio1:9000: Avg: 71.41 obj/s
 * http://minio2:9000: Avg: 46.40 obj/s
 * http://minio3:9000: Avg: 35.76 obj/s
 * http://minio4:9000: Avg: 35.86 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 213.14 obj/s
 * 50% Median: 189.33 obj/s
 * Slowest: 171.84 obj/s


──────────────────────────────────

Report: Total. Concurrency: 32. Ran: 4m57s
 * Average: 1515.27 MiB/s, 631.37 obj/s

Throughput by host:
 * http://minio1:9000: Avg: 571.86 MiB/s, 238.75 obj/s
 * http://minio2:9000: Avg: 378.12 MiB/s, 156.51 obj/s
 * http://minio3:9000: Avg: 284.71 MiB/s, 118.56 obj/s
 * http://minio4:9000: Avg: 280.39 MiB/s, 117.45 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 1596.4MiB/s, 673.66 obj/s
 * 50% Median: 1477.9MiB/s, 631.85 obj/s
 * Slowest: 1444.0MiB/s, 593.07 obj/s

4.4 100MiB 文件

  • get
1
2
3
4
5
6
7
8

warp get \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 100MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

Reqs: 4053, Errs:0, Objs:4053, Bytes: 395.80GiB
 -       GET Average: 14 Obj/s, 1350.1MiB/s; Current 13 Obj/s, 1250.6MiB/s, 2384.4 ms/req, TTFB: 38.2ms


Report: GET. Concurrency: 32. Ran: 4m58s
 * Average: 1351.21 MiB/s, 13.51 obj/s
 * Reqs: Avg: 2383.0ms, 50%: 1135.7ms, 90%: 5146.3ms, 99%: 5854.7ms, Fastest: 574.9ms, Slowest: 6330.3ms, StdDev: 1898.4ms
 * TTFB: Avg: 39ms, Best: 11ms, 25th: 29ms, Median: 31ms, 75th: 36ms, 90th: 47ms, 99th: 264ms, Worst: 462ms StdDev: 36ms

Throughput by host:
 * http://minio1:9000: Avg: 857.96 MiB/s, 8.58 obj/s
 * http://minio2:9000: Avg: 160.94 MiB/s, 1.61 obj/s
 * http://minio3:9000: Avg: 167.85 MiB/s, 1.68 obj/s
 * http://minio4:9000: Avg: 166.19 MiB/s, 1.66 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 1477.0MiB/s, 14.77 obj/s
 * 50% Median: 1348.3MiB/s, 13.48 obj/s
 * Slowest: 1265.3MiB/s, 12.65 obj/s
  • put
1
2
3
4
5
6
7
8

warp put \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 100MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

Reqs: 3420, Errs:0, Objs:3420, Bytes: 333.98GiB
 -       PUT Average: 11 Obj/s, 1145.2MiB/s; Current 8 Obj/s, 788.4MiB/s, 2781.0 ms/req


Report: PUT. Concurrency: 32. Ran: 4m58s
 * Average: 1147.93 MiB/s, 11.48 obj/s
 * Reqs: Avg: 2785.6ms, 50%: 2388.4ms, 90%: 4980.9ms, 99%: 7426.6ms, Fastest: 813.8ms, Slowest: 9144.3ms, StdDev: 1458.3ms

Throughput by host:
 * http://minio1:9000: Avg: 512.47 MiB/s, 5.12 obj/s
 * http://minio2:9000: Avg: 207.00 MiB/s, 2.07 obj/s
 * http://minio3:9000: Avg: 211.36 MiB/s, 2.11 obj/s
 * http://minio4:9000: Avg: 217.29 MiB/s, 2.17 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 1269.7MiB/s, 12.70 obj/s
 * 50% Median: 1148.6MiB/s, 11.49 obj/s
 * Slowest: 961.5MiB/s, 9.62 obj/s
  • mixed
1
2
3
4
5
6
7
8

warp mixed \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 100MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

Reqs: 8436, Errs:0, Objs:8436, Bytes: 493.75GiB
 -    DELETE Average: 3 Obj/s; Current 2 Obj/s, 34.8 ms/req
 -       GET Average: 13 Obj/s, 1262.4MiB/s; Current 11 Obj/s, 1097.6MiB/s, 2090.8 ms/req, TTFB: 38.2ms
 -       PUT Average: 4 Obj/s, 422.3MiB/s; Current 0 Obj/s, 25.2MiB/s, 891.0 ms/req
 -      STAT Average: 8 Obj/s; Current 3 Obj/s, 18.0 ms/req


Report: DELETE. Concurrency: 32. Ran: 4m57s
 * Average: 2.81 obj/s
 * Reqs: Avg: 37.1ms, 50%: 34.7ms, 90%: 52.9ms, 99%: 83.4ms, Fastest: 16.0ms, Slowest: 252.6ms, StdDev: 12.5ms

Throughput by host:
 * http://minio1:9000: Avg: 1.65 obj/s
 * http://minio2:9000: Avg: 0.39 obj/s
 * http://minio3:9000: Avg: 0.38 obj/s
 * http://minio4:9000: Avg: 0.43 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 9.18 obj/s
 * 50% Median: 2.34 obj/s
 * Slowest: 0.00 obj/s

──────────────────────────────────

Report: GET. Concurrency: 32. Ran: 4m58s
 * Average: 1265.11 MiB/s, 12.65 obj/s
 * Reqs: Avg: 2242.2ms, 50%: 1085.3ms, 90%: 4684.5ms, 99%: 5407.4ms, Fastest: 454.6ms, Slowest: 5991.3ms, StdDev: 1731.1ms
 * TTFB: Avg: 41ms, Best: 9ms, 25th: 28ms, Median: 31ms, 75th: 39ms, 90th: 50ms, 99th: 263ms, Worst: 468ms StdDev: 39ms

Throughput by host:
 * http://minio1:9000: Avg: 755.09 MiB/s, 7.55 obj/s
 * http://minio2:9000: Avg: 175.36 MiB/s, 1.75 obj/s
 * http://minio3:9000: Avg: 169.66 MiB/s, 1.70 obj/s
 * http://minio4:9000: Avg: 165.34 MiB/s, 1.65 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 1493.9MiB/s, 14.94 obj/s
 * 50% Median: 1279.1MiB/s, 12.79 obj/s
 * Slowest: 876.4MiB/s, 8.76 obj/s

──────────────────────────────────

Report: PUT. Concurrency: 32. Ran: 4m58s
 * Average: 422.25 MiB/s, 4.22 obj/s
 * Reqs: Avg: 830.9ms, 50%: 642.7ms, 90%: 1552.7ms, 99%: 1745.8ms, Fastest: 407.9ms, Slowest: 2059.4ms, StdDev: 429.9ms

Throughput by host:
 * http://minio1:9000: Avg: 261.31 MiB/s, 2.61 obj/s
 * http://minio2:9000: Avg: 51.72 MiB/s, 0.52 obj/s
 * http://minio3:9000: Avg: 49.89 MiB/s, 0.50 obj/s
 * http://minio4:9000: Avg: 60.25 MiB/s, 0.60 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 957.7MiB/s, 9.58 obj/s
 * 50% Median: 429.1MiB/s, 4.29 obj/s
 * Slowest: 0.00 obj/s

──────────────────────────────────

Report: STAT. Concurrency: 32. Ran: 4m57s
 * Average: 8.38 obj/s
 * Reqs: Avg: 20.1ms, 50%: 18.0ms, 90%: 25.1ms, 99%: 114.8ms, Fastest: 4.8ms, Slowest: 240.1ms, StdDev: 13.2ms

Throughput by host:
 * http://minio1:9000: Avg: 5.19 obj/s
 * http://minio2:9000: Avg: 1.20 obj/s
 * http://minio3:9000: Avg: 0.94 obj/s
 * http://minio4:9000: Avg: 1.12 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 18.83 obj/s
 * 50% Median: 8.00 obj/s
 * Slowest: 0.00 obj/s


──────────────────────────────────

Report: Total. Concurrency: 32. Ran: 4m58s
 * Average: 1687.36 MiB/s, 28.04 obj/s

Throughput by host:
 * http://minio1:9000: Avg: 1016.22 MiB/s, 16.98 obj/s
 * http://minio2:9000: Avg: 226.50 MiB/s, 3.85 obj/s
 * http://minio3:9000: Avg: 219.13 MiB/s, 3.48 obj/s
 * http://minio4:9000: Avg: 224.77 MiB/s, 3.78 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 1865.5MiB/s, 44.49 obj/s
 * 50% Median: 1584.7MiB/s, 27.85 obj/s
 * Slowest: 1373.9MiB/s, 17.74 obj/s

5. 10个数据位6个校验位

5.1 集群拓扑

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

●  minio1:9000
   Uptime: 57 seconds
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

●  minio2:9000
   Uptime: 24 seconds
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

●  minio3:9000
   Uptime: 54 seconds
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

●  minio4:9000
   Uptime: 21 seconds
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

┌──────┬───────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage          │ Erasure stripe size │ Erasure sets │
│ 1st  │ 0.5% (total: 7.3 TiB) │ 16                  │ 1            │
└──────┴───────────────────────┴─────────────────────┴──────────────┘

16 drives online, 0 drives offline, EC:6

5.2 4KiB 文件

  • get
1
2
3
4
5
6
7
8

warp get \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4KiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

Reqs: 3088828, Errs:0, Objs:3088828, Bytes: 11.78GiB
 -       GET Average: 10313 Obj/s, 40.3MiB/s; Current 10755 Obj/s, 42.0MiB/s, 2.9 ms/req, TTFB: 2.9ms


Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 40.29 MiB/s, 10314.68 obj/s
 * Reqs: Avg: 3.0ms, 50%: 2.5ms, 90%: 4.6ms, 99%: 9.5ms, Fastest: 1.6ms, Slowest: 218.6ms, StdDev: 1.8ms
 * TTFB: Avg: 3ms, Best: 2ms, 25th: 2ms, Median: 2ms, 75th: 3ms, 90th: 5ms, 99th: 9ms, Worst: 219ms StdDev: 2ms

Throughput by host:
 * http://minio1:9000: Avg: 10.63 MiB/s, 2722.38 obj/s
 * http://minio2:9000: Avg: 10.19 MiB/s, 2608.56 obj/s
 * http://minio3:9000: Avg: 9.72 MiB/s, 2487.66 obj/s
 * http://minio4:9000: Avg: 9.75 MiB/s, 2495.40 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 45.6MiB/s, 11665.60 obj/s
 * 50% Median: 41.3MiB/s, 10575.98 obj/s
 * Slowest: 26.9MiB/s, 6874.93 obj/s
  • put
1
2
3
4
5
6
7
8

warp put \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4KiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

Reqs: 1377457, Errs:0, Objs:1377457, Bytes: 5380.7MiB
 -       PUT Average: 4595 Obj/s, 18.0MiB/s; Current 4382 Obj/s, 17.1MiB/s, 7.2 ms/req


Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 17.95 MiB/s, 4595.44 obj/s
 * Reqs: Avg: 6.9ms, 50%: 6.1ms, 90%: 9.5ms, 99%: 21.1ms, Fastest: 2.7ms, Slowest: 106.4ms, StdDev: 3.3ms

Throughput by host:
 * http://minio1:9000: Avg: 4.54 MiB/s, 1161.00 obj/s
 * http://minio2:9000: Avg: 4.43 MiB/s, 1135.15 obj/s
 * http://minio3:9000: Avg: 4.48 MiB/s, 1146.89 obj/s
 * http://minio4:9000: Avg: 4.50 MiB/s, 1153.26 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 19.5MiB/s, 5000.30 obj/s
 * 50% Median: 18.1MiB/s, 4624.40 obj/s
 * Slowest: 15.2MiB/s, 3885.30 obj/s
  • mixed
1
2
3
4
5
6
7
8

warp mixed \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4KiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

Reqs: 2227623, Errs:0, Objs:2227623, Bytes: 5221.0MiB
 -    DELETE Average: 745 Obj/s; Current 603 Obj/s, 11.1 ms/req
 -       GET Average: 3350 Obj/s, 13.1MiB/s; Current 2724 Obj/s, 10.6MiB/s, 2.8 ms/req, TTFB: 2.8ms
 -       PUT Average: 1117 Obj/s, 4.4MiB/s; Current 909 Obj/s, 3.5MiB/s, 10.7 ms/req
 -      STAT Average: 2234 Obj/s; Current 1812 Obj/s, 2.4 ms/req


Report: DELETE. Concurrency: 32. Ran: 4m57s
 * Average: 744.22 obj/s
 * Reqs: Avg: 9.5ms, 50%: 5.6ms, 90%: 19.7ms, 99%: 56.2ms, Fastest: 3.6ms, Slowest: 209.6ms, StdDev: 10.5ms

Throughput by host:
 * http://minio1:9000: Avg: 193.43 obj/s
 * http://minio2:9000: Avg: 182.82 obj/s
 * http://minio3:9000: Avg: 182.76 obj/s
 * http://minio4:9000: Avg: 184.72 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 927.09 obj/s
 * 50% Median: 749.19 obj/s
 * Slowest: 557.12 obj/s

──────────────────────────────────

Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 13.08 MiB/s, 3349.09 obj/s
 * Reqs: Avg: 2.8ms, 50%: 2.5ms, 90%: 3.4ms, 99%: 10.3ms, Fastest: 1.6ms, Slowest: 217.5ms, StdDev: 1.8ms
 * TTFB: Avg: 3ms, Best: 2ms, 25th: 2ms, Median: 2ms, 75th: 3ms, 90th: 3ms, 99th: 10ms, Worst: 218ms StdDev: 2ms

Throughput by host:
 * http://minio1:9000: Avg: 3.39 MiB/s, 867.18 obj/s
 * http://minio2:9000: Avg: 3.23 MiB/s, 827.43 obj/s
 * http://minio3:9000: Avg: 3.19 MiB/s, 815.48 obj/s
 * http://minio4:9000: Avg: 3.27 MiB/s, 836.64 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 16.4MiB/s, 4185.61 obj/s
 * 50% Median: 13.2MiB/s, 3367.32 obj/s
 * Slowest: 9.6MiB/s, 2462.28 obj/s

──────────────────────────────────

Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 4.36 MiB/s, 1116.34 obj/s
 * Reqs: Avg: 8.9ms, 50%: 5.1ms, 90%: 18.5ms, 99%: 53.9ms, Fastest: 2.7ms, Slowest: 276.1ms, StdDev: 10.0ms

Throughput by host:
 * http://minio1:9000: Avg: 1.12 MiB/s, 285.84 obj/s
 * http://minio2:9000: Avg: 1.08 MiB/s, 275.65 obj/s
 * http://minio3:9000: Avg: 1.07 MiB/s, 273.49 obj/s
 * http://minio4:9000: Avg: 1.10 MiB/s, 280.57 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 5.5MiB/s, 1396.57 obj/s
 * 50% Median: 4.4MiB/s, 1122.22 obj/s
 * Slowest: 3.2MiB/s, 826.97 obj/s

──────────────────────────────────

Report: STAT. Concurrency: 32. Ran: 4m57s
 * Average: 2232.73 obj/s
 * Reqs: Avg: 2.5ms, 50%: 2.1ms, 90%: 3.1ms, 99%: 10.3ms, Fastest: 1.3ms, Slowest: 90.2ms, StdDev: 1.7ms

Throughput by host:
 * http://minio1:9000: Avg: 577.22 obj/s
 * http://minio2:9000: Avg: 551.27 obj/s
 * http://minio3:9000: Avg: 547.02 obj/s
 * http://minio4:9000: Avg: 555.64 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 2782.22 obj/s
 * 50% Median: 2248.00 obj/s
 * Slowest: 1657.38 obj/s


──────────────────────────────────

Report: Total. Concurrency: 32. Ran: 4m57s
 * Average: 17.44 MiB/s, 7442.39 obj/s

Throughput by host:
 * http://minio1:9000: Avg: 4.50 MiB/s, 1923.63 obj/s
 * http://minio2:9000: Avg: 4.31 MiB/s, 1837.15 obj/s
 * http://minio3:9000: Avg: 4.25 MiB/s, 1818.73 obj/s
 * http://minio4:9000: Avg: 4.36 MiB/s, 1857.47 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 21.8MiB/s, 9291.49 obj/s
 * 50% Median: 17.6MiB/s, 7484.74 obj/s
 * Slowest: 12.8MiB/s, 5503.75 obj/s

5.3 4MiB 文件

  • get
1
2
3
4
5
6
7
8

warp get \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

Reqs: 94255, Errs:0, Objs:94255, Bytes: 368.18GiB
 -       GET Average: 315 Obj/s, 1260.8MiB/s; Current 318 Obj/s, 1272.2MiB/s, 101.4 ms/req, TTFB: 31.4m


Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 1260.83 MiB/s, 315.21 obj/s
 * Reqs: Avg: 101.5ms, 50%: 97.0ms, 90%: 164.1ms, 99%: 317.5ms, Fastest: 24.9ms, Slowest: 730.4ms, StdDev: 57.1ms
 * TTFB: Avg: 31ms, Best: 12ms, 25th: 25ms, Median: 27ms, 75th: 29ms, 90th: 34ms, 99th: 233ms, Worst: 470ms StdDev: 27ms

Throughput by host:
 * http://minio1:9000: Avg: 536.43 MiB/s, 134.11 obj/s
 * http://minio2:9000: Avg: 247.00 MiB/s, 61.75 obj/s
 * http://minio3:9000: Avg: 237.86 MiB/s, 59.46 obj/s
 * http://minio4:9000: Avg: 239.47 MiB/s, 59.87 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 1312.2MiB/s, 328.04 obj/s
 * 50% Median: 1262.4MiB/s, 315.60 obj/s
 * Slowest: 1207.2MiB/s, 301.81 obj/s
  • put
1
2
3
4
5
6
7
8

warp put \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

Reqs: 77518, Errs:0, Objs:77518, Bytes: 302.80GiB
 -       PUT Average: 259 Obj/s, 1035.1MiB/s; Current 257 Obj/s, 1027.4MiB/s, 123.9 ms/req


Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 1035.13 MiB/s, 258.78 obj/s
 * Reqs: Avg: 123.6ms, 50%: 111.7ms, 90%: 198.5ms, 99%: 359.4ms, Fastest: 39.8ms, Slowest: 1074.3ms, StdDev: 62.4ms

Throughput by host:
 * http://minio1:9000: Avg: 406.34 MiB/s, 101.59 obj/s
 * http://minio2:9000: Avg: 208.35 MiB/s, 52.09 obj/s
 * http://minio3:9000: Avg: 225.90 MiB/s, 56.48 obj/s
 * http://minio4:9000: Avg: 194.52 MiB/s, 48.63 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 1055.3MiB/s, 263.84 obj/s
 * 50% Median: 1035.0MiB/s, 258.75 obj/s
 * Slowest: 1018.8MiB/s, 254.70 obj/s
  • mixed
1
2
3
4
5
6
7
8

warp mixed \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

Reqs: 186378, Errs:0, Objs:186378, Bytes: 436.81GiB
 -    DELETE Average: 62 Obj/s; Current 69 Obj/s, 26.9 ms/req
 -       GET Average: 280 Obj/s, 1118.9MiB/s; Current 280 Obj/s, 1121.0MiB/s, 67.0 ms/req, TTFB: 24.6ms
 -       PUT Average: 93 Obj/s, 373.0MiB/s; Current 95 Obj/s, 378.4MiB/s, 94.1 ms/req
 -      STAT Average: 186 Obj/s; Current 178 Obj/s, 15.2 ms/req


Report: DELETE. Concurrency: 32. Ran: 4m57s
 * Average: 62.18 obj/s
 * Reqs: Avg: 28.4ms, 50%: 26.4ms, 90%: 39.0ms, 99%: 68.2ms, Fastest: 9.8ms, Slowest: 253.8ms, StdDev: 11.7ms

Throughput by host:
 * http://minio1:9000: Avg: 24.47 obj/s
 * http://minio2:9000: Avg: 15.13 obj/s
 * http://minio3:9000: Avg: 11.26 obj/s
 * http://minio4:9000: Avg: 11.32 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 73.71 obj/s
 * 50% Median: 62.37 obj/s
 * Slowest: 50.56 obj/s

──────────────────────────────────

Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 1118.90 MiB/s, 279.73 obj/s
 * Reqs: Avg: 65.6ms, 50%: 64.8ms, 90%: 94.8ms, 99%: 270.1ms, Fastest: 13.2ms, Slowest: 794.5ms, StdDev: 38.2ms
 * TTFB: Avg: 24ms, Best: 6ms, 25th: 19ms, Median: 22ms, 75th: 25ms, 90th: 28ms, 99th: 117ms, Worst: 452ms StdDev: 21ms

Throughput by host:
 * http://minio1:9000: Avg: 425.13 MiB/s, 106.28 obj/s
 * http://minio2:9000: Avg: 276.41 MiB/s, 69.10 obj/s
 * http://minio3:9000: Avg: 211.35 MiB/s, 52.84 obj/s
 * http://minio4:9000: Avg: 205.57 MiB/s, 51.39 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 1186.2MiB/s, 296.54 obj/s
 * 50% Median: 1119.1MiB/s, 279.79 obj/s
 * Slowest: 1042.9MiB/s, 260.73 obj/s

──────────────────────────────────

Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 372.99 MiB/s, 93.25 obj/s
 * Reqs: Avg: 98.3ms, 50%: 73.5ms, 90%: 159.8ms, 99%: 266.6ms, Fastest: 38.2ms, Slowest: 938.9ms, StdDev: 50.1ms

Throughput by host:
 * http://minio1:9000: Avg: 140.11 MiB/s, 35.03 obj/s
 * http://minio2:9000: Avg: 91.97 MiB/s, 22.99 obj/s
 * http://minio3:9000: Avg: 68.97 MiB/s, 17.24 obj/s
 * http://minio4:9000: Avg: 71.90 MiB/s, 17.97 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 410.9MiB/s, 102.72 obj/s
 * 50% Median: 375.6MiB/s, 93.91 obj/s
 * Slowest: 323.8MiB/s, 80.94 obj/s

──────────────────────────────────

Report: STAT. Concurrency: 32. Ran: 4m57s
 * Average: 186.45 obj/s
 * Reqs: Avg: 14.6ms, 50%: 13.9ms, 90%: 19.6ms, 99%: 27.2ms, Fastest: 2.7ms, Slowest: 239.1ms, StdDev: 8.3ms

Throughput by host:
 * http://minio1:9000: Avg: 71.65 obj/s
 * http://minio2:9000: Avg: 45.56 obj/s
 * http://minio3:9000: Avg: 34.66 obj/s
 * http://minio4:9000: Avg: 34.62 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 207.46 obj/s
 * 50% Median: 186.15 obj/s
 * Slowest: 170.66 obj/s


──────────────────────────────────

Report: Total. Concurrency: 32. Ran: 4m57s
 * Average: 1491.89 MiB/s, 621.61 obj/s

Throughput by host:
 * http://minio1:9000: Avg: 565.17 MiB/s, 237.35 obj/s
 * http://minio2:9000: Avg: 368.35 MiB/s, 152.75 obj/s
 * http://minio3:9000: Avg: 280.30 MiB/s, 115.96 obj/s
 * http://minio4:9000: Avg: 277.44 MiB/s, 115.28 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 1575.1MiB/s, 665.68 obj/s
 * 50% Median: 1499.8MiB/s, 619.73 obj/s
 * Slowest: 1383.8MiB/s, 578.64 obj/s

5.4 100MiB 文件

  • get
1
2
3
4
5
6
7
8

warp get \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 100MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

Reqs: 4038, Errs:0, Objs:4038, Bytes: 394.34GiB
 -       GET Average: 13 Obj/s, 1345.7MiB/s; Current 13 Obj/s, 1270.3MiB/s, 2428.1 ms/req, TTFB: 38.1ms


Report: GET. Concurrency: 32. Ran: 4m58s
 * Average: 1346.85 MiB/s, 13.47 obj/s
 * Reqs: Avg: 2374.6ms, 50%: 1136.6ms, 90%: 4938.2ms, 99%: 5658.8ms, Fastest: 541.8ms, Slowest: 6260.1ms, StdDev: 1799.1ms
 * TTFB: Avg: 40ms, Best: 9ms, 25th: 29ms, Median: 31ms, 75th: 37ms, 90th: 46ms, 99th: 275ms, Worst: 471ms StdDev: 38ms

Throughput by host:
 * http://minio1:9000: Avg: 827.08 MiB/s, 8.27 obj/s
 * http://minio2:9000: Avg: 170.20 MiB/s, 1.70 obj/s
 * http://minio3:9000: Avg: 174.43 MiB/s, 1.74 obj/s
 * http://minio4:9000: Avg: 176.10 MiB/s, 1.76 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 1539.3MiB/s, 15.39 obj/s
 * 50% Median: 1347.7MiB/s, 13.48 obj/s
 * Slowest: 1274.4MiB/s, 12.74 obj/s
  • put
1
2
3
4
5
6
7
8

warp put \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 100MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

Reqs: 3173, Errs:0, Objs:3173, Bytes: 309.86GiB
 -       PUT Average: 11 Obj/s, 1061.8MiB/s; Current 8 Obj/s, 787.1MiB/s, 2965.1 ms/req


Report: PUT. Concurrency: 32. Ran: 4m58s
 * Average: 1063.68 MiB/s, 10.64 obj/s
 * Reqs: Avg: 3006.2ms, 50%: 2682.0ms, 90%: 5045.9ms, 99%: 7382.7ms, Fastest: 1099.2ms, Slowest: 10713.3ms, StdDev: 1368.8ms

Throughput by host:
 * http://minio1:9000: Avg: 435.55 MiB/s, 4.36 obj/s
 * http://minio2:9000: Avg: 207.85 MiB/s, 2.08 obj/s
 * http://minio3:9000: Avg: 218.78 MiB/s, 2.19 obj/s
 * http://minio4:9000: Avg: 201.80 MiB/s, 2.02 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 1192.9MiB/s, 11.93 obj/s
 * 50% Median: 1064.5MiB/s, 10.64 obj/s
 * Slowest: 892.9MiB/s, 8.93 obj/s
  • mixed
1
2
3
4
5
6
7
8

warp mixed \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 100MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

Reqs: 8280, Errs:0, Objs:8280, Bytes: 483.89GiB
 -    DELETE Average: 3 Obj/s; Current 2 Obj/s, 42.8 ms/req
 -       GET Average: 12 Obj/s, 1238.2MiB/s; Current 10 Obj/s, 975.5MiB/s, 2234.2 ms/req, TTFB: 42.1ms
 -       PUT Average: 4 Obj/s, 416.0MiB/s; Current 4 Obj/s, 371.2MiB/s, 917.7 ms/req
 -      STAT Average: 8 Obj/s; Current 12 Obj/s, 18.6 ms/req


Report: DELETE. Concurrency: 32. Ran: 4m57s
 * Average: 2.76 obj/s
 * Reqs: Avg: 39.2ms, 50%: 35.9ms, 90%: 57.6ms, 99%: 95.9ms, Fastest: 15.7ms, Slowest: 292.1ms, StdDev: 15.4ms

Throughput by host:
 * http://minio1:9000: Avg: 1.65 obj/s
 * http://minio2:9000: Avg: 0.42 obj/s
 * http://minio3:9000: Avg: 0.36 obj/s
 * http://minio4:9000: Avg: 0.36 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 9.00 obj/s
 * 50% Median: 2.00 obj/s
 * Slowest: 0.00 obj/s

──────────────────────────────────

Report: GET. Concurrency: 32. Ran: 4m59s
 * Average: 1240.36 MiB/s, 12.40 obj/s
 * Reqs: Avg: 2255.4ms, 50%: 1122.2ms, 90%: 4495.7ms, 99%: 5240.3ms, Fastest: 491.2ms, Slowest: 5973.4ms, StdDev: 1627.1ms
 * TTFB: Avg: 41ms, Best: 8ms, 25th: 29ms, Median: 32ms, 75th: 40ms, 90th: 51ms, 99th: 272ms, Worst: 482ms StdDev: 37ms

Throughput by host:
 * http://minio1:9000: Avg: 712.08 MiB/s, 7.12 obj/s
 * http://minio2:9000: Avg: 181.57 MiB/s, 1.82 obj/s
 * http://minio3:9000: Avg: 176.57 MiB/s, 1.77 obj/s
 * http://minio4:9000: Avg: 170.29 MiB/s, 1.70 obj/s

Throughput, split into 299 x 1s:
 * Fastest: 1589.3MiB/s, 15.89 obj/s
 * 50% Median: 1245.0MiB/s, 12.45 obj/s
 * Slowest: 871.4MiB/s, 8.71 obj/s

──────────────────────────────────

Report: PUT. Concurrency: 32. Ran: 4m58s
 * Average: 416.01 MiB/s, 4.16 obj/s
 * Reqs: Avg: 901.3ms, 50%: 735.7ms, 90%: 1621.3ms, 99%: 1853.5ms, Fastest: 403.7ms, Slowest: 2339.9ms, StdDev: 456.8ms

Throughput by host:
 * http://minio1:9000: Avg: 243.15 MiB/s, 2.43 obj/s
 * http://minio2:9000: Avg: 60.56 MiB/s, 0.61 obj/s
 * http://minio3:9000: Avg: 54.37 MiB/s, 0.54 obj/s
 * http://minio4:9000: Avg: 60.55 MiB/s, 0.61 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 875.9MiB/s, 8.76 obj/s
 * 50% Median: 426.3MiB/s, 4.26 obj/s
 * Slowest: 0.00 obj/s

──────────────────────────────────

Report: STAT. Concurrency: 32. Ran: 4m57s
 * Average: 8.25 obj/s
 * Reqs: Avg: 20.2ms, 50%: 18.4ms, 90%: 26.1ms, 99%: 106.1ms, Fastest: 5.6ms, Slowest: 258.3ms, StdDev: 11.6ms

Throughput by host:
 * http://minio1:9000: Avg: 4.89 obj/s
 * http://minio2:9000: Avg: 1.23 obj/s
 * http://minio3:9000: Avg: 1.08 obj/s
 * http://minio4:9000: Avg: 1.11 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 17.96 obj/s
 * 50% Median: 8.00 obj/s
 * Slowest: 0.00 obj/s


──────────────────────────────────

Report: Total. Concurrency: 32. Ran: 4m59s
 * Average: 1656.57 MiB/s, 27.57 obj/s

Throughput by host:
 * http://minio1:9000: Avg: 954.87 MiB/s, 16.08 obj/s
 * http://minio2:9000: Avg: 241.31 MiB/s, 4.04 obj/s
 * http://minio3:9000: Avg: 230.00 MiB/s, 3.74 obj/s
 * http://minio4:9000: Avg: 230.37 MiB/s, 3.76 obj/s

Throughput, split into 299 x 1s:
 * Fastest: 1865.5MiB/s, 39.79 obj/s
 * 50% Median: 1750.1MiB/s, 27.77 obj/s
 * Slowest: 1305.6MiB/s, 17.57 obj/s

6. 8个数据位8个校验位

6.1 集群拓扑

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

●  minio1:9000
   Uptime: 2 minutes
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

●  minio2:9000
   Uptime: 14 seconds
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

●  minio3:9000
   Uptime: 3 minutes
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

●  minio4:9000
   Uptime: 2 minutes
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

┌──────┬───────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage          │ Erasure stripe size │ Erasure sets │
│ 1st  │ 0.4% (total: 5.8 TiB) │ 16                  │ 1            │
└──────┴───────────────────────┴─────────────────────┴──────────────┘

16 drives online, 0 drives offline, EC:8

6.2 4KiB 文件

  • get
1
2
3
4
5
6
7
8

warp get \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4KiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

Reqs: 3181645, Errs:0, Objs:3181645, Bytes: 12.14GiB
 -       GET Average: 10632 Obj/s, 41.5MiB/s; Current 11162 Obj/s, 43.6MiB/s, 3.0 ms/req, TTFB: 2.9ms


Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 41.54 MiB/s, 10634.43 obj/s
 * Reqs: Avg: 2.9ms, 50%: 2.5ms, 90%: 3.9ms, 99%: 10.2ms, Fastest: 1.5ms, Slowest: 223.8ms, StdDev: 1.8ms
 * TTFB: Avg: 3ms, Best: 2ms, 25th: 2ms, Median: 2ms, 75th: 3ms, 90th: 4ms, 99th: 10ms, Worst: 224ms StdDev: 2ms

Throughput by host:
 * http://minio1:9000: Avg: 10.94 MiB/s, 2801.50 obj/s
 * http://minio2:9000: Avg: 10.36 MiB/s, 2651.45 obj/s
 * http://minio3:9000: Avg: 10.01 MiB/s, 2562.28 obj/s
 * http://minio4:9000: Avg: 10.20 MiB/s, 2610.45 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 47.3MiB/s, 12113.02 obj/s
 * 50% Median: 42.6MiB/s, 10909.83 obj/s
 * Slowest: 25.5MiB/s, 6519.31 obj/s
  • put
1
2
3
4
5
6
7
8

warp put \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4KiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

Reqs: 1416006, Errs:0, Objs:1416006, Bytes: 5531.3MiB
 -       PUT Average: 4724 Obj/s, 18.5MiB/s; Current 4524 Obj/s, 17.7MiB/s, 7.0 ms/req


Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 18.45 MiB/s, 4723.86 obj/s
 * Reqs: Avg: 6.8ms, 50%: 5.9ms, 90%: 9.4ms, 99%: 21.3ms, Fastest: 2.6ms, Slowest: 241.5ms, StdDev: 3.4ms

Throughput by host:
 * http://minio1:9000: Avg: 4.68 MiB/s, 1197.67 obj/s
 * http://minio2:9000: Avg: 4.57 MiB/s, 1171.16 obj/s
 * http://minio3:9000: Avg: 4.58 MiB/s, 1172.02 obj/s
 * http://minio4:9000: Avg: 4.62 MiB/s, 1183.99 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 20.3MiB/s, 5196.38 obj/s
 * 50% Median: 18.5MiB/s, 4724.29 obj/s
 * Slowest: 15.9MiB/s, 4078.18 obj/s
  • mixed
1
2
3
4
5
6
7
8

warp mixed \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4KiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

Reqs: 2130255, Errs:0, Objs:2130255, Bytes: 4992.8MiB
 -    DELETE Average: 712 Obj/s; Current 558 Obj/s, 10.8 ms/req
 -       GET Average: 3204 Obj/s, 12.5MiB/s; Current 2477 Obj/s, 9.7MiB/s, 2.9 ms/req, TTFB: 2.8ms
 -       PUT Average: 1068 Obj/s, 4.2MiB/s; Current 818 Obj/s, 3.2MiB/s, 10.5 ms/req
 -      STAT Average: 2136 Obj/s; Current 1642 Obj/s, 2.5 ms/req


Report: DELETE. Concurrency: 32. Ran: 4m57s
 * Average: 712.59 obj/s
 * Reqs: Avg: 10.3ms, 50%: 5.8ms, 90%: 22.1ms, 99%: 64.1ms, Fastest: 3.3ms, Slowest: 137.4ms, StdDev: 12.4ms

Throughput by host:
 * http://minio1:9000: Avg: 185.80 obj/s
 * http://minio2:9000: Avg: 176.54 obj/s
 * http://minio3:9000: Avg: 173.05 obj/s
 * http://minio4:9000: Avg: 177.17 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 905.84 obj/s
 * 50% Median: 704.76 obj/s
 * Slowest: 493.37 obj/s

──────────────────────────────────

Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 12.53 MiB/s, 3206.72 obj/s
 * Reqs: Avg: 2.8ms, 50%: 2.4ms, 90%: 3.6ms, 99%: 10.3ms, Fastest: 1.6ms, Slowest: 214.5ms, StdDev: 1.7ms
 * TTFB: Avg: 3ms, Best: 2ms, 25th: 2ms, Median: 2ms, 75th: 3ms, 90th: 4ms, 99th: 10ms, Worst: 214ms StdDev: 2ms

Throughput by host:
 * http://minio1:9000: Avg: 3.24 MiB/s, 829.76 obj/s
 * http://minio2:9000: Avg: 3.10 MiB/s, 793.88 obj/s
 * http://minio3:9000: Avg: 3.06 MiB/s, 782.50 obj/s
 * http://minio4:9000: Avg: 3.13 MiB/s, 800.85 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 16.0MiB/s, 4104.63 obj/s
 * 50% Median: 12.4MiB/s, 3172.08 obj/s
 * Slowest: 8.8MiB/s, 2246.27 obj/s

──────────────────────────────────

Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 4.18 MiB/s, 1068.89 obj/s
 * Reqs: Avg: 9.5ms, 50%: 5.3ms, 90%: 20.2ms, 99%: 60.1ms, Fastest: 2.8ms, Slowest: 225.2ms, StdDev: 11.3ms

Throughput by host:
 * http://minio1:9000: Avg: 1.08 MiB/s, 275.99 obj/s
 * http://minio2:9000: Avg: 1.03 MiB/s, 264.16 obj/s
 * http://minio3:9000: Avg: 1.02 MiB/s, 261.50 obj/s
 * http://minio4:9000: Avg: 1.04 MiB/s, 267.23 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 5.3MiB/s, 1361.08 obj/s
 * 50% Median: 4.1MiB/s, 1059.38 obj/s
 * Slowest: 2.9MiB/s, 747.06 obj/s

──────────────────────────────────

Report: STAT. Concurrency: 32. Ran: 4m57s
 * Average: 2137.78 obj/s
 * Reqs: Avg: 2.5ms, 50%: 2.1ms, 90%: 3.2ms, 99%: 10.4ms, Fastest: 1.3ms, Slowest: 74.4ms, StdDev: 1.7ms

Throughput by host:
 * http://minio1:9000: Avg: 550.75 obj/s
 * http://minio2:9000: Avg: 528.27 obj/s
 * http://minio3:9000: Avg: 524.15 obj/s
 * http://minio4:9000: Avg: 534.84 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 2735.38 obj/s
 * 50% Median: 2115.82 obj/s
 * Slowest: 1493.92 obj/s


──────────────────────────────────

Report: Total. Concurrency: 32. Ran: 4m57s
 * Average: 16.70 MiB/s, 7125.98 obj/s

Throughput by host:
 * http://minio1:9000: Avg: 4.32 MiB/s, 1842.09 obj/s
 * http://minio2:9000: Avg: 4.13 MiB/s, 1762.68 obj/s
 * http://minio3:9000: Avg: 4.08 MiB/s, 1741.02 obj/s
 * http://minio4:9000: Avg: 4.17 MiB/s, 1779.86 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 21.4MiB/s, 9106.94 obj/s
 * 50% Median: 16.6MiB/s, 7059.25 obj/s
 * Slowest: 11.7MiB/s, 4980.62 obj/s

6.3 4MiB 文件

  • get
1
2
3
4
5
6
7
8

warp get \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

Reqs: 94236, Errs:0, Objs:94236, Bytes: 368.11GiB
 -       GET Average: 314 Obj/s, 1256.1MiB/s; Current 305 Obj/s, 1218.7MiB/s, 102.4 ms/req, TTFB: 30.5m


Report: GET. Concurrency: 32. Ran: 4m58s
 * Average: 1256.14 MiB/s, 314.03 obj/s
 * Reqs: Avg: 101.9ms, 50%: 97.2ms, 90%: 162.1ms, 99%: 310.7ms, Fastest: 22.9ms, Slowest: 783.8ms, StdDev: 54.7ms
 * TTFB: Avg: 31ms, Best: 9ms, 25th: 25ms, Median: 27ms, 75th: 29ms, 90th: 34ms, 99th: 237ms, Worst: 479ms StdDev: 26ms

Throughput by host:
 * http://minio1:9000: Avg: 522.07 MiB/s, 130.52 obj/s
 * http://minio2:9000: Avg: 253.32 MiB/s, 63.33 obj/s
 * http://minio3:9000: Avg: 240.68 MiB/s, 60.17 obj/s
 * http://minio4:9000: Avg: 239.94 MiB/s, 59.98 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 1298.4MiB/s, 324.60 obj/s
 * 50% Median: 1257.0MiB/s, 314.25 obj/s
 * Slowest: 1153.8MiB/s, 288.44 obj/s
  • put
1
2
3
4
5
6
7
8

warp put \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

Reqs: 71312, Errs:0, Objs:71312, Bytes: 278.56GiB
 -       PUT Average: 238 Obj/s, 951.7MiB/s; Current 240 Obj/s, 959.1MiB/s, 133.8 ms/req


Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 951.70 MiB/s, 237.92 obj/s
 * Reqs: Avg: 134.4ms, 50%: 125.4ms, 90%: 193.8ms, 99%: 344.9ms, Fastest: 40.9ms, Slowest: 944.5ms, StdDev: 54.4ms

Throughput by host:
 * http://minio1:9000: Avg: 332.97 MiB/s, 83.24 obj/s
 * http://minio2:9000: Avg: 202.65 MiB/s, 50.66 obj/s
 * http://minio3:9000: Avg: 222.80 MiB/s, 55.70 obj/s
 * http://minio4:9000: Avg: 193.23 MiB/s, 48.31 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 992.4MiB/s, 248.10 obj/s
 * 50% Median: 950.8MiB/s, 237.70 obj/s
 * Slowest: 927.4MiB/s, 231.86 obj/s
  • mixed
1
2
3
4
5
6
7
8

warp mixed \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128

Reqs: 71312, Errs:0, Objs:71312, Bytes: 278.56GiB
 -       PUT Average: 238 Obj/s, 951.7MiB/s; Current 240 Obj/s, 959.1MiB/s, 133.8 ms/req


Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 951.70 MiB/s, 237.92 obj/s
 * Reqs: Avg: 134.4ms, 50%: 125.4ms, 90%: 193.8ms, 99%: 344.9ms, Fastest: 40.9ms, Slowest: 944.5ms, StdDev: 54.4ms

Throughput by host:
 * http://minio1:9000: Avg: 332.97 MiB/s, 83.24 obj/s
 * http://minio2:9000: Avg: 202.65 MiB/s, 50.66 obj/s
 * http://minio3:9000: Avg: 222.80 MiB/s, 55.70 obj/s
 * http://minio4:9000: Avg: 193.23 MiB/s, 48.31 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 992.4MiB/s, 248.10 obj/s
 * 50% Median: 950.8MiB/s, 237.70 obj/s
 * Slowest: 927.4MiB/s, 231.86 obj/s



Cleanup
Cleanup Done                                                                    root@hw-ai-k8s-v100-03:root@hw-ai-k8s-v100-03:~# warp mixed \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m
╭─────────────────────────────────╮
│ WARP S3 Benchmark Tool by MinIO │
╰─────────────────────────────────╯

Benchmarking: Press 'q' to stop benchmark. ...

 λ █████████████████████████████████████████████████████████████████████████ 100%

Reqs: 180976, Errs:0, Objs:180976, Bytes: 424.14GiB
 -    DELETE Average: 61 Obj/s; Current 64 Obj/s, 27.3 ms/req
 -       GET Average: 273 Obj/s, 1090.1MiB/s; Current 273 Obj/s, 1092.6MiB/s, 67.6 ms/req, TTFB: 24.9ms
 -       PUT Average: 91 Obj/s, 363.5MiB/s; Current 97 Obj/s, 387.5MiB/s, 103.5 ms/req
 -      STAT Average: 182 Obj/s; Current 167 Obj/s, 15.1 ms/req


Report: DELETE. Concurrency: 32. Ran: 4m57s
 * Average: 60.55 obj/s
 * Reqs: Avg: 28.2ms, 50%: 26.5ms, 90%: 36.3ms, 99%: 62.7ms, Fastest: 10.2ms, Slowest: 266.1ms, StdDev: 11.3ms

Throughput by host:
 * http://minio1:9000: Avg: 23.12 obj/s
 * http://minio2:9000: Avg: 15.24 obj/s
 * http://minio3:9000: Avg: 11.08 obj/s
 * http://minio4:9000: Avg: 11.15 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 73.27 obj/s
 * 50% Median: 60.72 obj/s
 * Slowest: 47.91 obj/s

──────────────────────────────────

Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 1090.19 MiB/s, 272.55 obj/s
 * Reqs: Avg: 67.2ms, 50%: 66.2ms, 90%: 97.0ms, 99%: 268.0ms, Fastest: 14.7ms, Slowest: 706.5ms, StdDev: 37.0ms
 * TTFB: Avg: 24ms, Best: 7ms, 25th: 20ms, Median: 22ms, 75th: 25ms, 90th: 28ms, 99th: 65ms, Worst: 473ms StdDev: 20ms

Throughput by host:
 * http://minio1:9000: Avg: 410.30 MiB/s, 102.57 obj/s
 * http://minio2:9000: Avg: 273.21 MiB/s, 68.30 obj/s
 * http://minio3:9000: Avg: 205.05 MiB/s, 51.26 obj/s
 * http://minio4:9000: Avg: 201.91 MiB/s, 50.48 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 1155.7MiB/s, 288.92 obj/s
 * 50% Median: 1091.2MiB/s, 272.80 obj/s
 * Slowest: 1024.8MiB/s, 256.20 obj/s

──────────────────────────────────

Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 363.44 MiB/s, 90.86 obj/s
 * Reqs: Avg: 101.5ms, 50%: 74.6ms, 90%: 164.3ms, 99%: 278.5ms, Fastest: 40.9ms, Slowest: 819.7ms, StdDev: 52.0ms

Throughput by host:
 * http://minio1:9000: Avg: 136.38 MiB/s, 34.10 obj/s
 * http://minio2:9000: Avg: 90.70 MiB/s, 22.67 obj/s
 * http://minio3:9000: Avg: 67.75 MiB/s, 16.94 obj/s
 * http://minio4:9000: Avg: 68.67 MiB/s, 17.17 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 414.3MiB/s, 103.59 obj/s
 * 50% Median: 360.2MiB/s, 90.05 obj/s
 * Slowest: 329.3MiB/s, 82.33 obj/s

──────────────────────────────────

Report: STAT. Concurrency: 32. Ran: 4m57s
 * Average: 181.69 obj/s
 * Reqs: Avg: 15.0ms, 50%: 14.3ms, 90%: 19.9ms, 99%: 27.7ms, Fastest: 3.4ms, Slowest: 237.9ms, StdDev: 8.1ms

Throughput by host:
 * http://minio1:9000: Avg: 69.58 obj/s
 * http://minio2:9000: Avg: 44.80 obj/s
 * http://minio3:9000: Avg: 33.47 obj/s
 * http://minio4:9000: Avg: 33.91 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 202.19 obj/s
 * 50% Median: 181.09 obj/s
 * Slowest: 161.25 obj/s


──────────────────────────────────

Report: Total. Concurrency: 32. Ran: 4m57s
 * Average: 1453.63 MiB/s, 605.65 obj/s

Throughput by host:
 * http://minio1:9000: Avg: 546.67 MiB/s, 229.35 obj/s
 * http://minio2:9000: Avg: 363.91 MiB/s, 151.01 obj/s
 * http://minio3:9000: Avg: 272.76 MiB/s, 112.72 obj/s
 * http://minio4:9000: Avg: 270.56 MiB/s, 112.69 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 1538.9MiB/s, 654.57 obj/s
 * 50% Median: 1473.0MiB/s, 606.32 obj/s
 * Slowest: 1371.0MiB/s, 562.78 obj/s

6.4 100MiB 文件

  • get
1
2
3
4
5
6
7
8

warp get \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 100MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

Reqs: 3995, Errs:0, Objs:3995, Bytes: 390.14GiB
 -       GET Average: 13 Obj/s, 1329.9MiB/s; Current 12 Obj/s, 1220.1MiB/s, 2318.8 ms/req, TTFB: 38.0ms


Report: GET. Concurrency: 32. Ran: 4m58s
 * Average: 1330.99 MiB/s, 13.31 obj/s
 * Reqs: Avg: 2416.1ms, 50%: 1268.1ms, 90%: 4682.1ms, 99%: 5328.5ms, Fastest: 584.5ms, Slowest: 5890.1ms, StdDev: 1648.1ms
 * TTFB: Avg: 38ms, Best: 9ms, 25th: 29ms, Median: 31ms, 75th: 36ms, 90th: 50ms, 99th: 239ms, Worst: 454ms StdDev: 32ms

Throughput by host:
 * http://minio1:9000: Avg: 774.20 MiB/s, 7.74 obj/s
 * http://minio2:9000: Avg: 180.97 MiB/s, 1.81 obj/s
 * http://minio3:9000: Avg: 187.91 MiB/s, 1.88 obj/s
 * http://minio4:9000: Avg: 188.56 MiB/s, 1.89 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 1433.3MiB/s, 14.33 obj/s
 * 50% Median: 1331.2MiB/s, 13.31 obj/s
 * Slowest: 1252.5MiB/s, 12.53 obj/s
  • put
1
2
3
4
5
6
7
8

warp put \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 100MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

Reqs: 2864, Errs:0, Objs:2864, Bytes: 279.69GiB
 -       PUT Average: 10 Obj/s, 959.9MiB/s; Current 5 Obj/s, 459.0MiB/s, 3273.6 ms/req


Report: PUT. Concurrency: 32. Ran: 4m58s
 * Average: 963.17 MiB/s, 9.63 obj/s
 * Reqs: Avg: 3319.8ms, 50%: 2995.1ms, 90%: 5336.0ms, 99%: 7491.0ms, Fastest: 1002.8ms, Slowest: 8713.4ms, StdDev: 1362.2ms

Throughput by host:
 * http://minio1:9000: Avg: 368.11 MiB/s, 3.68 obj/s
 * http://minio2:9000: Avg: 206.17 MiB/s, 2.06 obj/s
 * http://minio3:9000: Avg: 196.50 MiB/s, 1.96 obj/s
 * http://minio4:9000: Avg: 192.80 MiB/s, 1.93 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 1085.4MiB/s, 10.85 obj/s
 * 50% Median: 962.5MiB/s, 9.62 obj/s
 * Slowest: 783.6MiB/s, 7.84 obj/s
  • mixed
1
2
3
4
5
6
7
8

warp mixed \
  --host minio1:9000,minio2:9000,minio3:9000,minio4:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 100MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

Reqs: 8003, Errs:0, Objs:8003, Bytes: 468.07GiB
 -    DELETE Average: 3 Obj/s; Current 3 Obj/s, 41.4 ms/req
 -       GET Average: 12 Obj/s, 1199.1MiB/s; Current 10 Obj/s, 982.1MiB/s, 2327.1 ms/req, TTFB: 42.6ms
 -       PUT Average: 4 Obj/s, 400.4MiB/s; Current 2 Obj/s, 152.7MiB/s, 949.5 ms/req
 -      STAT Average: 8 Obj/s; Current 10 Obj/s, 20.6 ms/req


Report: DELETE. Concurrency: 32. Ran: 4m56s
 * Average: 2.65 obj/s
 * Reqs: Avg: 39.0ms, 50%: 35.7ms, 90%: 56.7ms, 99%: 97.3ms, Fastest: 10.5ms, Slowest: 257.3ms, StdDev: 15.5ms

Throughput by host:
 * http://minio1:9000: Avg: 1.45 obj/s
 * http://minio2:9000: Avg: 0.45 obj/s
 * http://minio3:9000: Avg: 0.38 obj/s
 * http://minio4:9000: Avg: 0.40 obj/s

Throughput, split into 296 x 1s:
 * Fastest: 9.00 obj/s
 * 50% Median: 2.00 obj/s
 * Slowest: 0.00 obj/s

──────────────────────────────────

Report: GET. Concurrency: 32. Ran: 4m58s
 * Average: 1201.56 MiB/s, 12.02 obj/s
 * Reqs: Avg: 2319.4ms, 50%: 1442.2ms, 90%: 4318.5ms, 99%: 5118.0ms, Fastest: 513.9ms, Slowest: 6131.3ms, StdDev: 1492.4ms
 * TTFB: Avg: 44ms, Best: 10ms, 25th: 29ms, Median: 33ms, 75th: 42ms, 90th: 61ms, 99th: 300ms, Worst: 1.05s StdDev: 42ms

Throughput by host:
 * http://minio1:9000: Avg: 651.95 MiB/s, 6.52 obj/s
 * http://minio2:9000: Avg: 191.48 MiB/s, 1.91 obj/s
 * http://minio3:9000: Avg: 179.45 MiB/s, 1.79 obj/s
 * http://minio4:9000: Avg: 178.80 MiB/s, 1.79 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 1493.8MiB/s, 14.94 obj/s
 * 50% Median: 1207.2MiB/s, 12.07 obj/s
 * Slowest: 862.5MiB/s, 8.63 obj/s

──────────────────────────────────

Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 400.41 MiB/s, 4.00 obj/s
 * Reqs: Avg: 1000.8ms, 50%: 850.3ms, 90%: 1682.8ms, 99%: 1965.6ms, Fastest: 416.9ms, Slowest: 2763.9ms, StdDev: 447.4ms

Throughput by host:
 * http://minio1:9000: Avg: 215.02 MiB/s, 2.15 obj/s
 * http://minio2:9000: Avg: 63.66 MiB/s, 0.64 obj/s
 * http://minio3:9000: Avg: 63.06 MiB/s, 0.63 obj/s
 * http://minio4:9000: Avg: 61.72 MiB/s, 0.62 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 854.8MiB/s, 8.55 obj/s
 * 50% Median: 416.6MiB/s, 4.17 obj/s
 * Slowest: 0.00 obj/s

──────────────────────────────────

Report: STAT. Concurrency: 32. Ran: 4m57s
 * Average: 7.98 obj/s
 * Reqs: Avg: 21.1ms, 50%: 19.5ms, 90%: 27.3ms, 99%: 121.7ms, Fastest: 4.3ms, Slowest: 252.6ms, StdDev: 13.6ms

Throughput by host:
 * http://minio1:9000: Avg: 4.26 obj/s
 * http://minio2:9000: Avg: 1.38 obj/s
 * http://minio3:9000: Avg: 1.12 obj/s
 * http://minio4:9000: Avg: 1.28 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 18.00 obj/s
 * 50% Median: 8.00 obj/s
 * Slowest: 0.00 obj/s


──────────────────────────────────

Report: Total. Concurrency: 32. Ran: 4m58s
 * Average: 1600.81 MiB/s, 26.63 obj/s

Throughput by host:
 * http://minio1:9000: Avg: 865.50 MiB/s, 14.34 obj/s
 * http://minio2:9000: Avg: 254.86 MiB/s, 4.37 obj/s
 * http://minio3:9000: Avg: 240.48 MiB/s, 3.89 obj/s
 * http://minio4:9000: Avg: 239.83 MiB/s, 4.07 obj/s

Throughput, split into 298 x 1s:
 * Fastest: 1698.0MiB/s, 39.98 obj/s
 * 50% Median: 1492.2MiB/s, 26.92 obj/s
 * Slowest: 1587.9MiB/s, 16.88 obj/s

7. 总结

7.1 4KiB 文件

数据位/校验位操作吞吐量
2/2get54.29 MiB/s
2/2put36.61 MiB/s
12/4get38.16 MiB/s
12/4put17.95 MiB/s
10/6get40.29 MiB/s
10/6put17.95 MiB/s
8/8get41.54 MiB/s
8/8put18.45 MiB/s

7.2 4MB 文件

数据位/校验位操作吞吐量
2/2get1228.90 MiB/s
2/2put998.27 MiB/s
12/4get1261.87 MiB/s
12/4put1117.66 MiB/s
10/6get1260.83 MiB/s
10/6put1035.13 MiB/s
8/8get1256.14 MiB/s
8/8put951.70 MiB/s

7.3 100MB 文件

数据位/校验位操作吞吐量
2/2get1246.97 MiB/s
2/2put1034.00 MiB/s
12/4get1351.21 MiB/s
12/4put1147.93 MiB/s
10/6get1346.85 MiB/s
10/6put1063.68 MiB/s
8/8get1330.99 MiB/s
8/8put963.17 MiB/s

7.4 小结

  • 提高并发 32 到 64 或文件大小增加到 1GB 后，吞吐量没有明显提升，甚至略微下降
  • 4KiB 文件测试时，磁盘 io 是瓶颈，Io util 接近 85% 以上，CPU 占用明显增加能达到 40Core 以上
  • 4MiB+ 文件测试时，网卡占用很大，基本满载，接近 10Gbps
  • minio 不适合存储小文件，要尽量保持在 4MiB 以上
  • 在网卡受限的情况下，纠删配置对读取吞吐影响不大，对写入吞吐有一定影响
微信公众号
作者
微信公众号

相关内容