MinIO 调优

1. 硬件

1.1 CPU

支持 AVX/AVX-512，用于纠删码计算、SIMD 操作
支持 AES-NI，用于加密对象

单机存储总量	推荐的 vCPU 数
最多 1 TB	8
最多 10 TB	16
最多 100 TB	32
最多 1 PB	64
大于 1 PB	128

1.2 内存

尽可能大一点能提高并发，增加 Page Cache 命中率。

磁盘数量	32 GiB	64 GiB 内存	128 GiB 内存	256 GiB 内存	512 GiB 内存
4 个磁盘	1,074	2,149	4,297	8,595	17,190
8 个磁盘	840	1,680	3,361	6,722	13,443
16 个磁盘	585	1,170	2,341	4,681	9,362

1.3 硬盘

推荐采用多机多盘，单机硬盘数量在 4、8、16，2 的幂次
直接提供裸盘，不要使用 RAID
磁盘的规格尽可能一致，不要有慢盘，否则会有严重木桶效应

1.4 网络

以顺序读取 HDD 200 MB/s，NVMe 3000 MB/s 为例，计算可支持的硬盘数量：

网络带宽	吞吐上限	HDD 数量 (200MB/s)	NVMe 数量 (3000MB/s)
1 GbE	125 MB/s	0.6	0.04
10 GbE	1.25 GB/s	6	0.4
25 GbE	3.125 GB/s	15	1
50 GbE	6.25 GB/s	31	2
100 GbE	12.5 GB/s	62	4

如果是多机部署，还需要考虑机器之间的数据传输，需要按照机器数量适当增加网卡。

隔离存储网络和业务网络也是推荐的做法，需要多增加网络设备。

2. 内核配置

2.1 网络优化

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
cat << 'EOF' | sudo tee /etc/sysctl.d/99-minio-network.conf
# 增加核心缓冲区最大值 (64 MiB)
net.core.rmem_max=67108864
net.core.wmem_max=67108864

# 增加 TCP 缓冲区范围
net.ipv4.tcp_rmem=4096 131072 67108864
net.ipv4.tcp_wmem=4096 131072 67108864

# 启用 TCP 窗口缩放 (必须开启，否则无法超过 64KB)
net.ipv4.tcp_window_scaling=1

# 提高并发连接建立能力
net.core.somaxconn=65535
net.ipv4.tcp_max_syn_backlog=65535

# 优化高带宽网络下的丢包处理
net.core.netdev_max_backlog=250000

# 开启 TCP 快速开启 (TFO)，减少握手延迟
net.ipv4.tcp_fastopen=3

# 禁用空闲后的慢启动（保持高吞吐状态）
net.ipv4.tcp_slow_start_after_idle=0
EOF
sysctl -p /etc/sysctl.d/99-minio-network.conf

2.2 禁用交换分区

1
2
3
4
cat << 'EOF' | sudo tee /etc/sysctl.d/99-minio-vm.conf
vm.swappiness=10
EOF
sysctl -p /etc/sysctl.d/99-minio-vm.conf

3. 系统配置

3.1 设置 TSC 时间同步

TSC 时间同步比系统时间同步更准确，更稳定。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
cat >/etc/systemd/system/clocksource-tsc.service <<'EOF'
[Unit]
Description=Set clocksource to tsc
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource'

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable clocksource-tsc.service
systemctl start clocksource-tsc.service

4. 文件系统

1
mkfs.xfs -f /dev/nvme1n1

1
mount -o noatime,nodiratime,logbufs=8,logbsize=256k,max_retries=0 /dev/nvme1n1 /mnt/data1

使用 xfs 格式，更适合大文件、高并发
logbufs=8 提高元数据并发写入能力
logbsize=256k 增加日志缓冲区大小
noatime 禁用文件访问时间
nodiratime 禁用目录访问时间
max_retries=0 禁用重试，避免与 MinIO 的自动重试冲突

5. 对比

5.1 部署方式

一台单机四盘 NVMe 部署 MinIO，一台通过 IPoIB 进行测试。

5.2 优化前

挂载硬盘

1
2
3
for i in {0..3}; do
    mount /dev/nvme${i}n1 /mnt/data${i}
done

启动参考

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
export CONTAINER_CLI=nerdctl
export IMAGE=registry.cn-beijing.aliyuncs.com/opshub/minio-minio:RELEASE.2025-04-22T22-12-26Z
export ROOT_USER=minioadmin
export ROOT_PASSWORD=minioadmin
export MINIO_ERASURE_SET_DRIVE_COUNT=4
export MINIO_STORAGE_CLASS_STANDARD=EC:1
export POOL_0="http://minioib2/mnt/data{0...3}"

$CONTAINER_CLI run -d \
  --net host \
  --ulimit nofile=1048576:1048576 \
  --name minio \
  -v /mnt/data0:/mnt/data0 \
  -v /mnt/data1:/mnt/data1 \
  -v /mnt/data2:/mnt/data2 \
  -v /mnt/data3:/mnt/data3 \
  -e "MINIO_ROOT_USER=$ROOT_USER" \
  -e "MINIO_ROOT_PASSWORD=$ROOT_PASSWORD" \
  -e "MINIO_ERASURE_SET_DRIVE_COUNT=$MINIO_ERASURE_SET_DRIVE_COUNT" \
  -e "MINIO_STORAGE_CLASS_STANDARD=$MINIO_STORAGE_CLASS_STANDARD" \
  $IMAGE server \
  $POOL_0 \
  --console-address ":9090" \
  --address ":9000"

测试读取

1
2
3
4
5
6
7
8
warp get \
  --host minioib2:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
Reqs: 583656, Errs:0, Objs:583656, Bytes: 2279.91GiB
 -       GET Average: 1946 Obj/s, 7782.3MiB/s; Current 1923 Obj/s, 7693.7MiB/s, 16.4 ms/req, TTFB: 2.2ms


Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 7782.25 MiB/s, 1945.56 obj/s
 * Reqs: Avg: 16.4ms, 50%: 13.3ms, 90%: 28.7ms, 99%: 50.2ms, Fastest: 4.7ms, Slowest: 281.0ms, StdDev: 9.3ms
 * TTFB: Avg: 2ms, Best: 1ms, 25th: 2ms, Median: 2ms, 75th: 3ms, 90th: 4ms, 99th: 6ms, Worst: 56ms StdDev: 1ms

Throughput, split into 297 x 1s:
 * Fastest: 8276.5MiB/s, 2069.13 obj/s
 * 50% Median: 7824.0MiB/s, 1955.99 obj/s
 * Slowest: 6538.6MiB/s, 1634.65 obj/s

测试写入

1
2
3
4
5
6
7
8
warp put \
  --host minioib2:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
Reqs: 128689, Errs:0, Objs:128689, Bytes: 502.69GiB
 -       PUT Average: 430 Obj/s, 1721.4MiB/s; Current 476 Obj/s, 1902.4MiB/s, 73.4 ms/req


Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 1721.44 MiB/s, 430.36 obj/s
 * Reqs: Avg: 74.4ms, 50%: 73.4ms, 90%: 82.6ms, 99%: 105.9ms, Fastest: 28.5ms, Slowest: 297.0ms, StdDev: 8.5ms

Throughput, split into 297 x 1s:
 * Fastest: 1907.9MiB/s, 476.97 obj/s
 * 50% Median: 1724.8MiB/s, 431.20 obj/s
 * Slowest: 1567.6MiB/s, 391.89 obj/s

5.3 优化后

挂载硬盘

1
2
3
4
for i in {0..3}; do
  mount -t xfs /dev/nvme${i}n1 /mnt/data${i} \
  -o noatime,nodiratime,logbufs=8,logbsize=256k
done

启动参考

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
export CONTAINER_CLI=nerdctl
export IMAGE=registry.cn-beijing.aliyuncs.com/opshub/minio-minio:RELEASE.2025-04-22T22-12-26Z
export ROOT_USER=minioadmin
export ROOT_PASSWORD=minioadmin
export MINIO_ERASURE_SET_DRIVE_COUNT=4
export MINIO_STORAGE_CLASS_STANDARD=EC:1
export POOL_0="http://minioib2/mnt/data{0...3}"

$CONTAINER_CLI run -d \
  --net host \
  --cap-add=IPC_LOCK \
  --oom-score-adj=-1000 \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  --ulimit nofile=1048576:1048576 \
  --memory-swappiness=0 \
  --shm-size=1gb  \
  --name minio \
  -v /mnt/data0:/mnt/data0 \
  -v /mnt/data1:/mnt/data1 \
  -v /mnt/data2:/mnt/data2 \
  -v /mnt/data3:/mnt/data3 \
  -e "MINIO_ROOT_USER=$ROOT_USER" \
  -e "MINIO_ROOT_PASSWORD=$ROOT_PASSWORD" \
  -e "MINIO_ERASURE_SET_DRIVE_COUNT=$MINIO_ERASURE_SET_DRIVE_COUNT" \
  -e "MINIO_STORAGE_CLASS_STANDARD=$MINIO_STORAGE_CLASS_STANDARD" \
  $IMAGE server \
  $POOL_0 \
  --console-address ":9090" \
  --address ":9000"

测试读取

1
2
3
4
5
6
7
8
warp get \
  --host minioib2:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
Reqs: 603065, Errs:0, Objs:603065, Bytes: 2355.72GiB
 -       GET Average: 2015 Obj/s, 8061.8MiB/s; Current 1905 Obj/s, 7620.9MiB/s, 15.7 ms/req, TTFB: 2.2ms


Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 8062.56 MiB/s, 2015.64 obj/s
 * Reqs: Avg: 15.9ms, 50%: 12.8ms, 90%: 28.1ms, 99%: 48.8ms, Fastest: 4.7ms, Slowest: 293.7ms, StdDev: 9.1ms
 * TTFB: Avg: 2ms, Best: 1ms, 25th: 2ms, Median: 2ms, 75th: 3ms, 90th: 3ms, 99th: 6ms, Worst: 44ms StdDev: 1ms

Throughput, split into 297 x 1s:
 * Fastest: 8456.9MiB/s, 2114.21 obj/s
 * 50% Median: 8075.6MiB/s, 2018.90 obj/s
 * Slowest: 7376.6MiB/s, 1844.14 obj/s

测试写入

1
2
3
4
5
6
7
8
warp put \
  --host minioib2:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
Reqs: 130873, Errs:0, Objs:130873, Bytes: 511.22GiB
 -       PUT Average: 437 Obj/s, 1749.3MiB/s; Current 456 Obj/s, 1824.8MiB/s, 73.7 ms/req


Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 1749.09 MiB/s, 437.27 obj/s
 * Reqs: Avg: 73.2ms, 50%: 72.0ms, 90%: 81.5ms, 99%: 105.0ms, Fastest: 27.1ms, Slowest: 299.0ms, StdDev: 9.1ms

Throughput, split into 297 x 1s:
 * Fastest: 1928.3MiB/s, 482.06 obj/s
 * 50% Median: 1754.0MiB/s, 438.50 obj/s
 * Slowest: 1519.2MiB/s, 379.81 obj/s

5.4 小结

前后	GET	PUT
优化前	7782.25 MiB/s	1721.44 MiB/s
优化后	8062.56 MiB/s	1749.09 MiB/s

优化后读取吞吐量提升 3.6%，写入吞吐量提升 1.6%