This page looks best with JavaScript enabled

MinIO Multi-Node, Multi-Disk Deployment and Operations

 ·  ☕ 13 min read

1. Environment Preparation

1.1 Preparing Data Disks

  • Inspect the data disks
1

lsblk -d -o NAME,SIZE,TYPE | grep nvme

1
2
3
4

nvme0n1 745.2G disk
nvme1n1 745.2G disk
nvme2n1 745.2G disk
nvme3n1 745.2G disk
  • Prepare the storage directories
1
2
3

for i in {0..3}; do
    mkdir -p /mnt/data${i}
done
  • Format the data disks
1
2
3

for i in {0..3}; do
    mkfs.xfs -f /dev/nvme${i}n1
done
  • Mount the data disks
1
2
3

for i in {0..3}; do
    mount /dev/nvme${i}n1 /mnt/data${i}
done
  • Clear the data disks
1
2
3
4

for i in {0..3}; do
  rm -rf /mnt/data${i}/*
  rm -rf /mnt/data${i}/.minio.sys
done
  • Check the mount status
1

df -h | grep "/mnt"

1
2
3
4

/dev/nvme0n1                   745G  5.3G  740G   1% /mnt/data0
/dev/nvme1n1                   745G  5.3G  740G   1% /mnt/data1
/dev/nvme2n1                   745G  5.3G  740G   1% /mnt/data2
/dev/nvme3n1                   745G  5.3G  740G   1% /mnt/data3

1.2 Time Synchronization

1
2

apt install -y chrony
systemctl enable --now chrony

2. Deploying MinIO

Each node has 4 disks; we deploy two machines to form a single minio cluster.

2.1 Configuring hosts

1
2
3
4

cat >> /etc/hosts <<EOF
10.0.0.1 minio1
10.0.0.2 minio2
EOF

Configuring hosts pins the node names, which makes operations easier and avoids identification problems during expansion.

2.1 Environment Variables

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

export CONTAINER_CLI=nerdctl
export IMAGE=minio/minio:RELEASE.2025-04-22T22-12-26Z

export ROOT_USER=minioadmin
export ROOT_PASSWORD=minioadmin

export MINIO_ERASURE_SET_DRIVE_COUNT=4
export MINIO_STORAGE_CLASS_STANDARD=EC:2

export POOL_0="http://minio{1...2}/mnt/data{0...3}"

Web consoles in minio versions after RELEASE.2025-04-22T22-12-26Z are extremely bare-bones: you can only create buckets, and you cannot configure policies, keys, and so on. Complex configuration requires the command line.

MINIO_ERASURE_SET_DRIVE_COUNT is the size of an erasure set and cannot exceed 16.
MINIO_STORAGE_CLASS_STANDARD is the redundancy level of the erasure code and cannot exceed erasure set size/2.

A single deployment has just one pool, provided to minio on one line at startup.

2.2 Starting minio

The startup command is identical on every node; here we start two nodes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

$CONTAINER_CLI run -d \
  --security-opt apparmor=unconfined \
  --security-opt seccomp=unconfined \
  --net host \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  --ulimit nofile=1048576:1048576 \
  --memory-swappiness=0 \
  --name minio \
  -v /mnt/data0:/mnt/data0 \
  -v /mnt/data1:/mnt/data1 \
  -v /mnt/data2:/mnt/data2 \
  -v /mnt/data3:/mnt/data3 \
  -e "MINIO_ROOT_USER=$ROOT_USER" \
  -e "MINIO_ROOT_PASSWORD=$ROOT_PASSWORD" \
  -e "MINIO_ERASURE_SET_DRIVE_COUNT=$MINIO_ERASURE_SET_DRIVE_COUNT" \
  -e "MINIO_STORAGE_CLASS_STANDARD=$MINIO_STORAGE_CLASS_STANDARD" \
  $IMAGE server \
  $POOL_0 \
  --console-address ":9090" \
  --address ":9000"

9090 is the MinIO console port and 9000 is the MinIO API port; use port 9000 when accessing data.

After one node starts, it waits for the other nodes to join the cluster.

3. Cluster Management

3.1 Configuring Credentials

  • Install mc
1
2
3

wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
mv mc /usr/bin/
  • Configure credentials
1

mc alias set local http://127.0.0.1:9000 minioadmin minioadmin
  • View the cluster
1

mc admin info local

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

●  minio1:9000
   Uptime: 46 seconds
   Version: 2025-04-22T22:12:26Z
   Network: 2/2 OK
   Drives: 4/4 OK
   Pool: 1

●  minio2:9000
   Uptime: 42 seconds
   Version: 2025-04-22T22:12:26Z
   Network: 2/2 OK
   Drives: 4/4 OK
   Pool: 1

┌──────┬───────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage          │ Erasure stripe size │ Erasure sets │
│ 1st  │ 0.7% (total: 2.9 TiB) │ 4                   │ 2            │
└──────┴───────────────────────┴─────────────────────┴──────────────┘

8 drives online, 0 drives offline, EC:2

Erasure stripe size indicates the number of disks per erasure set, and Erasure sets indicates the number of erasure sets. Here we have 4-disk erasure sets, 2 erasure sets, EC:2 with two parity disks, giving a usable capacity ratio of 50%.

3.2 Bucket Management

  • Create a bucket
1

mc mb local/test
  • View bucket details
1

mc stat local/test
  • List buckets
1

mc ls local
  • Delete
1

mc rb local/test

3.3 Users and Policies

  • Create a user
1

mc admin user add local myuser mypassword
  • View a user
1

mc admin user info local myuser
  • Attach a built-in policy
1

mc admin policy attach local readwrite --user myuser

Built-in policies apply to all buckets. The available values are: readwrite, writeonly, consoleAdmin, diagnostics, readonly.

  • Remove user permissions
1

mc admin policy detach local readwrite --user myuser
  • Delete a policy
1

mc admin policy rm local mypolicy
  • Custom policy
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

cat > test-readwrite.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::test"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::test/*"
      ]
    }
  ]
}
EOF

1

mc admin policy create local test-readwrite test-readwrite.json
  • Attach a custom policy
1

mc admin policy attach local test-readwrite --user myuser

3.4 File Management

  • Copy a file
1

mc cp ./test.txt local/test/test.txt
  • Copy a directory
1

mc cp -r ./ local/test/
  • View file details
1

mc stat local/test/test.txt

3.5 rebalance

minio rebalance is only effective on multi-node clusters.

  • Start rebalance
1

mc admin rebalance start local
  • View rebalance status
1

mc admin rebalance status local
  • Stop rebalance
1

mc admin rebalance stop local

3.6 Bucket Versioning

  • Enable bucket versioning
1

mc version enable local/test2
  • View bucket versioning
1

mc version info local/test2
  • Copy files
1

mc cp -r ./ local/test2/
  • View file versions
1

mc ls --versions local/test2/

1
2

5B STANDARD aa673997-a590-4b04-b3be-80627fa3352d v2 PUT bbbb
5B STANDARD f5bcc176-3544-432f-924c-eb654a317ca0 v1 PUT bbbb
  • Restore a specific version of a file
1

mc cp --version-id f5bcc176-3544-432f-924c-eb654a317ca0 local/test2/bbbb ./bbbb-v1

3.6 Statistics Details

  • Bucket usage
1

mc stat local/test2

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

mc stat local/test2
Name      : test2
Size      : N/A
Type      : folder

Properties:
  Versioning: Enabled
  Location: us-east-1
  Anonymous: Disabled
  ILM: Disabled

Usage:
      Total size: 1002 MiB
   Objects count: 1,652
  Versions count: 2,967

Object sizes histogram:
    509 object(s) BETWEEN_1024B_AND_1_MB
    423 object(s) BETWEEN_1024_B_AND_64_KB
      9 object(s) BETWEEN_10_MB_AND_64_MB
  • File usage
1

mc stat local/test2/bbbb

1
2
3
4
5
6
7

Name      : bbbb
Size      : 5 B
ETag      : 9448a1bf333fadd2a57965ec38487b89
VersionID : aa673997-a590-4b04-b3be-80627fa3352d
Type      : file
Metadata  :
  Content-Type: application/octet-stream

3.6 Backing Up format.json

format.json is minio’s format file, recording the cluster’s configuration information. It is recommended to back it up every time you adjust the cluster topology.

1
2
3
4
5

mkdir -p /backup/miniosys/
for i in {0..3}; do
    mkdir -p /backup/miniosys/data${i}
    cp /mnt/data${i}/.minio.sys/format.json /backup/miniosys/data${i}/format.json
done

View the backup

1

ls -l /backup/miniosys/

1
2
3
4

drwxr-xr-x 2 root root 4096 Feb  6 19:40 data0
drwxr-xr-x 2 root root 4096 Feb  6 19:40 data1
drwxr-xr-x 2 root root 4096 Feb  6 19:40 data2
drwxr-xr-x 2 root root 4096 Feb  6 19:40 data3

4 Backing Up Data

4.1 Full Backup with mc

1

mc cp -r ./ local/backup/

cp uploads everything in full every time rather than incrementally.

1

mc mirror ./ local/backup/ --remove --disable-multipart --monitoring-address 0.0.0.0:9001

mirror uploads incrementally; with remove, files deleted at the source are also deleted from the bucket.

4.2 Incremental Backup with restic

  • Install restic
1

apt install -y restic
  • Set environment variables
1
2
3
4

export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export RESTIC_REPOSITORY=s3:http://127.0.0.1:9000/backup
export RESTIC_PASSWORD="backuppassword"

restic requires a backup password to be set.

  • Initialize
1

restic init
  • Back up data
1

restic backup ./

restic backs up incrementally.

  • View backup status
1

restic snapshots
  • Restore data
1

restic restore <snapshot_ID> --target ./restore

1

du -sh ./restore
  • Keep 365 days of backups
1

restic forget --keep-last 365 --prune

4.3 Incremental Backup with rclone

  • Install rclone
1

apt install -y rclone
  • Configure rclone
1
2
3
4
5
6

rclone config create minio s3 \
  access_key_id minioadmin \
  secret_access_key minioadmin \
  endpoint http://127.0.0.1:9000 \
  region us-east-1 \
  force_path_style true
  • Back up data
1

rclone sync ./ minio:backup --progress

rclone backs up incrementally.

  • Back up overwrites and deletions to a separate directory
1
2
3

rclone sync ./ minio:backup \
  --backup-dir minio:backupold/$(date +%Y%m%d_%H%M%S) \
  --progress

Now overwritten or deleted files are backed up under the backupold bucket.

  • Restore data
1

rclone copy minio:backup ./backup_restore --progress

5. Node Restart

  • Stop one minio node
1

$CONTAINER_CLI stop minio
  • View the cluster status

Note that the endpoint configured in local here needs to be switched to a node that is running normally.

1

mc admin info local

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

●  minio1:9000
   Uptime: 2 hours
   Version: 2025-04-22T22:12:26Z
   Network: 1/2 OK
   Drives: 4/4 OK
   Pool: 1

●  minio2:9000
   Uptime: offline
   Drives: 0/4 OK

┌──────┬───────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage          │ Erasure stripe size │ Erasure sets │
│ 1st  │ 0.8% (total: 1.5 TiB) │ 4                   │ 2            │
└──────┴───────────────────────┴─────────────────────┴──────────────┘

1.5 GiB Used, 2 Buckets, 186 Objects
1 node offline, 4 drives online, 4 drives offline, EC:2

Data can only be accessed normally if the number of online disks meets the requirement; here one disk is missing, so data cannot be accessed normally.

  • Adjust parameters and restart the minio node

The number of data blocks and parity blocks cannot be adjusted; other parameters can.

  • View the cluster status
1

mc admin info local

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

●  minio1:9000
   Uptime: 2 hours
   Version: 2025-04-22T22:12:26Z
   Network: 2/2 OK
   Drives: 4/4 OK
   Pool: 1

●  minio2:9000
   Uptime: 59 seconds
   Version: 2025-04-22T22:12:26Z
   Network: 2/2 OK
   Drives: 4/4 OK
   Pool: 1

┌──────┬───────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage          │ Erasure stripe size │ Erasure sets │
│ 1st  │ 0.8% (total: 2.9 TiB) │ 4                   │ 2            │
└──────┴───────────────────────┴─────────────────────┴──────────────┘

1.5 GiB Used, 2 Buckets, 186 Objects
8 drives online, 0 drives offline, EC:2
  • Access data
1

mc cp -r local/test/ ./recreate/

6. Adding Nodes

  • Expansion constraints

The newly added nodes form a new pool, which does not interfere with the original pool.

Do not add just one node; add at least two at a time to guarantee the data reliability of the new pool.

The erasure code parameters of the new nodes must match those of the existing nodes, and the disk count must be an integer multiple of the erasure set size.

During expansion, you need to stop minio and modify the startup parameters of all instances to be consistent.

  • Configure hosts
1
2
3
4
5
6

cat >> /etc/hosts <<EOF
10.0.0.1 minio1
10.0.0.2 minio2
10.0.0.3 minio3
10.0.0.14 minio4
EOF
  • Set environment variables
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

export CONTAINER_CLI=nerdctl
export IMAGE=minio/minio:RELEASE.2025-04-22T22-12-26Z

export ROOT_USER=minioadmin
export ROOT_PASSWORD=minioadmin

export MINIO_ERASURE_SET_DRIVE_COUNT=4
export MINIO_STORAGE_CLASS_STANDARD=EC:2

export POOL_0="http://minio{1...2}/mnt/data{0...3}"
export POOL_1="http://minio{3...4}/mnt/data{0...3}"
  • Delete minio on the old nodes
1

$CONTAINER_CLI rm -f minio
  • Start the new nodes and update all the old nodes
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

$CONTAINER_CLI run -d \
  --security-opt apparmor=unconfined \
  --security-opt seccomp=unconfined \
  --net host \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  --ulimit nofile=1048576:1048576 \
  --memory-swappiness=0 \
  --name minio \
  -v /mnt/data0:/mnt/data0 \
  -v /mnt/data1:/mnt/data1 \
  -v /mnt/data2:/mnt/data2 \
  -v /mnt/data3:/mnt/data3 \
  -e "MINIO_ROOT_USER=$ROOT_USER" \
  -e "MINIO_ROOT_PASSWORD=$ROOT_PASSWORD" \
  -e "MINIO_ERASURE_SET_DRIVE_COUNT=$MINIO_ERASURE_SET_DRIVE_COUNT" \
  -e "MINIO_STORAGE_CLASS_STANDARD=$MINIO_STORAGE_CLASS_STANDARD" \
  $IMAGE server \
  $POOL_0 \
  $POOL_1 \
  --console-address ":9090" \
  --address ":9000"
  • Wait for startup
1

mc admin info local

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

●  minio1:9000
   Uptime: 1 minute
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

●  minio2:9000
   Uptime: 2 minutes
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 1

●  minio3:9000
   Uptime: 3 minutes
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 2

●  minio4:9000
   Uptime: 3 minutes
   Version: 2025-04-22T22:12:26Z
   Network: 4/4 OK
   Drives: 4/4 OK
   Pool: 2

┌──────┬────────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage           │ Erasure stripe size │ Erasure sets │
│ 1st  │ 0.9% (total: 2.9 TiB)  │ 4                   │ 2            │
│ 2nd  │ 26.2% (total: 3.5 TiB) │ 4                   │ 2            │
└──────┴────────────────────────┴─────────────────────┴──────────────┘

3.8 GiB Used, 3 Buckets, 16,022 Objects
16 drives online, 0 drives offline, EC:2

At this point you will see two pools: 1st is the previous pool, and 2nd is the newly added pool.

  • Test writing new data
1
2
3
4
5
6
7

┌──────┬────────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage           │ Erasure stripe size │ Erasure sets │
│ 1st  │ 1.1% (total: 2.9 TiB)  │ 4                   │ 2            │
│ 2nd  │ 26.6% (total: 3.5 TiB) │ 4                   │ 2            │
└──────┴────────────────────────┴─────────────────────┴──────────────┘

13 GiB Used, 3 Buckets, 18,692 Objects

New data is written more to the new pool, while the old pool also receives some writes.

  • rebalance test
1

mc admin rebalance start local
  • View rebalance status
1

mc admin rebalance status local

1
2
3
4
5
6
7
8

Per-pool usage:
┌────────┬────────┐
│ Pool-0 │ Pool-1 │
│ 1.24%  │ 26.23% │
└────────┴────────┘
Summary:
Data: 10 GiB (1684 objects, 1684 versions)
Time: 1m3.22950215s (0s to completion)

minio balances both within a pool and between pools.

7. Replacing a Single Disk

After replacing a single disk, keep the mount directory unchanged.

  • Stop the minio service
1

$CONTAINER_CLI stop minio
  • Mount the new disk

Reformat and mount the new disk to the original directory.

1
2

rm -rf /mnt/data0/*
rm -rf /mnt/data0/.minio.sys
  • Start the minio node
1

$CONTAINER_CLI start minio
  • View the cluster status
1

mc admin heal local

1

No active healing is detected for new disks, though 1 offline disk(s) found.

1
2
3
4

mc admin heal local
Objects Healed: 16, 56 KiB (53.4%)
Objects Failed: 0
Heal rate: 363 obj/s, 1.2 MiB/s

minio rebuilds the data automatically.

8. Node Rebuild

Four nodes form two pools. If the data of one node is deleted, one of the pools will not have enough replicas, and it can only be read from, not written to.

  • Stop the minio service
1

$CONTAINER_CLI stop minio
  • Delete the minio data
1
2
3
4

for i in {0..3}; do
  rm -rf /mnt/data${i}/*
  rm -rf /mnt/data${i}/.minio.sys
done
  • View the cluster status
1

mc admin info local

1
2
3

●  minio4:9000
   Uptime: offline
   Drives: 0/4 OK

The node status is offline.

  • Start the minio node
1

$CONTAINER_CLI start minio
  • View the data rebuild status
1

mc admin heal local --verbose

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

 mc admin heal local --verbose
Servers status:
==============
Pool 1st:
  minio1:9000:
  +  /mnt/data0 : HEALING
  |__   Progress: 90%
  |__    Started: now
  |__   Capacity: 5.2 GiB/745 GiB
  +  /mnt/data1 : HEALING
  |__   Progress: 90%
  |__    Started: now
  |__   Capacity: 5.2 GiB/745 GiB
  +  /mnt/data2 : HEALING
  |__   Progress: 95%
  |__    Started: now
  |__   Capacity: 5.2 GiB/745 GiB
  +  /mnt/data3 : HEALING
  |__   Progress: 95%
  |__    Started: now
  |__   Capacity: 5.2 GiB/745 GiB

Pool 2nd:

Summary:
=======
Objects Healed: 4, 856 B (90.4%)
Objects Failed: 0
Heal rate: 142 obj/s, 30 KiB/s

2 of 4 sets exceeds reduced parity count EC:1 lost/offline disks

9. Common Errors

  • Error on the old nodes during expansion
1
2
3

FATAL Unable to initialize backend:
/mnt/data0 drive is already being used in another erasure deployment.
(Number of drives specified: 16 but the number of drives found in the 5th drive's format.json: 8)

If you run into this problem, there is currently no fix other than backing up and rebuilding the cluster. This is because minio tries to form all disks into a new pool rather than adding a new pool. It is recommended to use hosts to pin node names when deploying.

微信公众号
WRITTEN BY
微信公众号

See Also