1. Hardware

1.1 CPU

• Support for AVX/AVX-512, used for erasure coding computation and SIMD operations
• Support for AES-NI, used for object encryption

1.2 Memory

More is better where possible, since it raises concurrency and increases the Page Cache hit rate.

1.3 Disk

• Multi-node, multi-disk is recommended, with 4, 8, or 16 disks per node — powers of two
• Provide raw disks directly; do not use RAID
• Keep disk specifications as uniform as possible; a single slow disk causes a severe barrel effect

1.4 Network

Taking sequential reads of HDD 200 MB/s and NVMe 3000 MB/s as an example, calculate the number of disks that can be supported:

For a multi-node deployment, you also need to account for data transfer between machines, so add NICs in proportion to the number of machines.

Isolating the storage network from the business network is also a recommended practice, and it requires additional network devices.

2. Kernel Configuration

2.1 Network Optimization

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

cat << 'EOF' | sudo tee /etc/sysctl.d/99-minio-network.conf
# 增加核心缓冲区最大值 (64 MiB)
net.core.rmem_max=67108864
net.core.wmem_max=67108864

# 增加 TCP 缓冲区范围
net.ipv4.tcp_rmem=4096 131072 67108864
net.ipv4.tcp_wmem=4096 131072 67108864

# 启用 TCP 窗口缩放 (必须开启，否则无法超过 64KB)
net.ipv4.tcp_window_scaling=1

# 提高并发连接建立能力
net.core.somaxconn=65535
net.ipv4.tcp_max_syn_backlog=65535

# 优化高带宽网络下的丢包处理
net.core.netdev_max_backlog=250000

# 开启 TCP 快速开启 (TFO)，减少握手延迟
net.ipv4.tcp_fastopen=3

# 禁用空闲后的慢启动（保持高吞吐状态）
net.ipv4.tcp_slow_start_after_idle=0
EOF
sysctl -p /etc/sysctl.d/99-minio-network.conf

2.2 Disable Swap

1
2
3
4

cat << 'EOF' | sudo tee /etc/sysctl.d/99-minio-vm.conf
vm.swappiness=10
EOF
sysctl -p /etc/sysctl.d/99-minio-vm.conf

3. System Configuration

3.1 Set Up TSC Time Synchronization

TSC time synchronization is more accurate and more stable than system time synchronization.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

cat >/etc/systemd/system/clocksource-tsc.service <<'EOF'
[Unit]
Description=Set clocksource to tsc
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource'

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable clocksource-tsc.service
systemctl start clocksource-tsc.service

4. File System

1

mkfs.xfs -f /dev/nvme1n1

1

mount -o noatime,nodiratime,logbufs=8,logbsize=256k,max_retries=0 /dev/nvme1n1 /mnt/data1

• Use the xfs format — better suited to large files and high concurrency
• logbufs=8 improves concurrent metadata write capability
• logbsize=256k increases the log buffer size
• noatime disables file access time
• nodiratime disables directory access time
• max_retries=0 disables retries, avoiding conflicts with MinIO’s own automatic retries

5. Choosing an Erasure Coding Scheme

M (Data Blocks), K (Parity Blocks)

Basic principles:

• The system bottleneck is disk IO
• The more data blocks there are, the less redundancy and the faster the system
• The more parity blocks there are, the more failed disks can be tolerated
• minio distributes each block (M+K) evenly across every node
• At least M blocks must be healthy to read successfully, and at least M+1 blocks must be healthy to write successfully

If you can draw the data block distribution diagram, making the choice should be easy.

6. Choosing a File Size

blockSizeV2 is defined in the source at https://github.com/minio/minio/blob/7aac2a2c5b7c882e68c1ce017d8256be2feea27f/cmd/object-api-common.go#L37. It is fixed at 1M (10M in older versions) and represents the processing window size for data, mainly affecting CPU, MEM, and small-file processing speed.

The size of the data file is a separate matter. Under an EC 10:6 configuration, minio writes a 1GB file by splitting the 1GB file into 10 data blocks, then computing 6 parity blocks, and storing each block in an erasure set chosen to be as spread out as possible. Each block is stored as a Part data file and a Meta metadata file.

Note the following:

• If a data block is only at the KB level, minio writes the data directly into the Meta file for storage
• For large file transfers, streaming chunking is used, and minio stores each chunk separately according to its size

Therefore, to achieve greater throughput, we need to:

• Use larger chunks when streaming uploads, or use –disable-multipart to disable chunking
• Keep stored files from being too small; the threshold is around 1GB for HDD, and you should test it yourself

7. Monitoring Dashboards

minio has four built-in dimensions of monitoring metrics: cluster, node, bucket, and resource. Running the following commands generates the corresponding scrape configuration.

• Cluster

1

mc admin prometheus generate local

• Node

1

mc admin prometheus generate local node

• Bucket

1

mc admin prometheus generate local bucket

• Resource

1

mc admin prometheus generate local resource

• Grafana dashboards

https://github.com/minio/minio/tree/master/docs/metrics/prometheus/grafana