XFS Operations Under RAID

The xfs filesystem does not support shrinking. This post is mainly about disk replacement and expansion under RAID 10.

1. RAID States and Disk States

In mdadm, the RAID states are:

clean, normal
degraded, still readable and writable, but with reduced fault tolerance
reshaping, reorganization in progress, appears during expansion and shrinking
resyncing, data synchronization in progress
recovering, data recovery in progress, accompanied by degradation as a disk is rebuilt
failed

In mdadm, the disk states are:

active, participates in data reads and writes
spare, a hot spare disk that automatically takes over for a failed disk
failed, a failed disk
sync, synchronizing, a normal state
rebuilding
set-x, all disks under the same marker form one mirror group

2. RAID 0

2.1 Creating RAID 0

Stop md0

1
2
umount /mnt/mdadmdm
mdadm --stop /dev/md0

Create RAID 0

1
mdadm --create --verbose /dev/md0 --level=0 --raid-devices=4 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1

Check RAID status

1
mdadm --detail /dev/md0

1
2
3
4
5
    State : clean
       0     259        0        0      active sync   /dev/nvme0n1
       1     259        1        1      active sync   /dev/nvme1n1
       2     259        4        2      active sync   /dev/nvme2n1
       3     259        3        3      active sync   /dev/nvme3n1

Format the RAID

1
mkfs.xfs -f /dev/md0

Mount the RAID

1
2
mkdir -p /mnt/mdadm
mount /dev/md0 /mnt/mdadm

Check disk size

1
df -h /mnt/mdadm/

1
2
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        3.0T   21G  2.9T   1% /mnt/mdadm

3. RAID 1

3.1 Creating RAID 1

Stop md0

1
2
umount /mnt/mdadm
mdadm --stop /dev/md0

Create RAID 1

1
mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/nvme0n1 /dev/nvme1n1

Format the RAID

1
mkfs.xfs -f /dev/md0

Mount the RAID

1
2
mkdir -p /mnt/mdadm
mount /dev/md0 /mnt/mdadm

Check disk size

1
df -h /mnt/mdadm/

1
2
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        745G  5.3G  740G   1% /mnt/mdadm

3.2 Single-Disk Replacement

Wipe the new disk

1
wipefs -a /dev/nvme2n1

Add a hot spare

1
mdadm /dev/md0 --add /dev/nvme2n1

Mark the failed disk

1
mdadm /dev/md0 --fail /dev/nvme1n1

Remove the failed disk

1
mdadm /dev/md0 --remove /dev/nvme1n1

Check status

1
mdadm --detail /dev/md0

1
2
3
4
    State : clean, degraded, recovering
    Number   Major   Minor   RaidDevice State
       0     259        0        0      active sync   /dev/nvme0n1
       2     259        4        1      spare rebuilding   /dev/nvme2n1

3.3 Adding a Disk to Expand

Add one disk to the array

1
mdadm /dev/md0 --add /dev/nvme3n1

Expand RAID 1

A newly added disk is a hot spare by default; the hot spare must be marked as a data disk.

1
mdadm /dev/md0 --grow --raid-devices=3

Check status

1
mdadm --detail /dev/md0

1
2
3
4
5
    State : clean, degraded, recovering
    Number   Major   Minor   RaidDevice State
       0     259        0        0      active sync   /dev/nvme0n1
       2     259        4        1      spare rebuilding   /dev/nvme2n1
       3     259        3        2      spare rebuilding   /dev/nvme3n1

Because every disk in RAID 1 holds the same data, there is no balancing or expansion operation.

4. RAID 10

4.1 Creating RAID 10

Stop md0

1
2
umount /mnt/mdadm
mdadm --stop /dev/md0

Create the RAID

1
mdadm --create --verbose /dev/md0 --level=10 --raid-devices=4 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1

Check RAID status

1
mdadm --detail /dev/md0

1
2
3
4
5
    Number   Major   Minor   RaidDevice State
       0     259        0        0      active sync set-A   /dev/nvme0n1
       1     259        1        1      active sync set-B   /dev/nvme1n1
       2     259        4        2      active sync set-A   /dev/nvme2n1
       3     259        3        3      active sync set-B   /dev/nvme3n1

Format the RAID

1
mkfs.xfs -f /dev/md0

Mount the RAID

1
2
mkdir -p /mnt/mdadm
mount /dev/md0 /mnt/mdadm

Check disk size

1
df -h /mnt/mdadm/

1
2
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        1.5T   11G  1.5T   1% /mnt/mdadm

Check disk size

1
df -h /mnt/mdadm/

1
2
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        1.5T   61G  1.4T   5% /mnt/mdadm

A 50G file 4m_seq_write.0.0 was written.

4.2 Single-Disk Replacement

A hot spare can be used for automatic failover or to replace a healthy data disk.

Add the hot spare to the array

1
mdadm /dev/md0 --add /dev/nvme4n1 /dev/nvme5n1

Remove the hot spare

1
mdadm /dev/md0 --remove /dev/nvme5n1

Check status (the hot spare shows as spare)

1
mdadm --detail /dev/md0

1
2
3
4
5
6
7
    Number   Major   Minor   RaidDevice State
       0     259        0        0      active sync set-A   /dev/nvme0n1
       1     259        1        1      active sync set-B   /dev/nvme1n1
       2     259        4        2      active sync set-A   /dev/nvme2n1
       3     259        3        3      active sync set-B   /dev/nvme3n1

       4     259        2        -      spare   /dev/nvme4n1

Kick the data disk out

1
mdadm /dev/md0 --fail /dev/nvme3n1

Check status

1
mdadm --detail /dev/md0

1
2
3
4
5
6
7
8
    State : clean, degraded, recovering
    Number   Major   Minor   RaidDevice State
       0     259        0        0      active sync set-A   /dev/nvme0n1
       1     259        1        1      active sync set-B   /dev/nvme1n1
       2     259        4        2      active sync set-A   /dev/nvme2n1
       4     259        2        3      spare rebuilding   /dev/nvme4n1

       3     259        3        -      faulty   /dev/nvme3n1

At this point the array is degraded and rebuilding data.

Remove the failed disk

1
mdadm /dev/md0 --remove /dev/nvme3n1

4.3 Adding Disks to Expand

Add two disks to the array

1
mdadm /dev/md0 --add /dev/nvme3n1 /dev/nvme5n1

Check status (the new disks show as spare)

1
mdadm --detail /dev/md0

1
2
3
4
5
6
7
8
9
    State : clean
    Number   Major   Minor   RaidDevice State
       0     259        0        0      active sync set-A   /dev/nvme0n1
       1     259        1        1      active sync set-B   /dev/nvme1n1
       2     259        4        2      active sync set-A   /dev/nvme2n1
       4     259        2        3      active sync set-B   /dev/nvme4n1

       3     259        3        -      spare   /dev/nvme3n1
       5     259        5        -      spare   /dev/nvme5n1

Newly added disks are hot spares by default.

Expand RAID 10

1
mdadm /dev/md0 --grow --raid-devices=6

Check status

1
mdadm --detail /dev/md0

1
2
3
4
5
6
7
8
    State : clean, reshaping
    Number   Major   Minor   RaidDevice State
       0     259        0        0      active sync set-A   /dev/nvme0n1
       1     259        1        1      active sync set-B   /dev/nvme1n1
       2     259        4        2      active sync set-A   /dev/nvme2n1
       4     259        2        3      active sync set-B   /dev/nvme4n1
       5     259        5        4      active sync set-A   /dev/nvme5n1
       3     259        3        5      active sync set-B   /dev/nvme3n1

Wait until the reshaping state finishes; cat /proc/mdstat shows how much longer it will take.

Expand the filesystem

1
xfs_growfs /mnt/mdadm

Check disk size

1
df -h /mnt/mdadm/

1
2
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        2.2T   66G  2.2T   3% /mnt/mdadm

The filesystem expansion is complete.

5. RAID 6

5.1 Creating RAID 6

Stop md0

1
2
umount /mnt/mdadm
mdadm --stop /dev/md0

Create RAID 6

1
mdadm --create --verbose /dev/md0 --level=6 --raid-devices=4 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1

Format the RAID

1
mkfs.xfs -f /dev/md0

Mount the RAID

1
2
mkdir -p /mnt/mdadm
mount /dev/md0 /mnt/mdadm

Check disk size

1
df -h /mnt/mdadm/

1
2
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        1.5T   11G  1.5T   1% /mnt/mdadm

5.2 Single-Disk Replacement

Wipe the new disk

1
wipefs -a /dev/nvme4n1

Add a hot spare

1
mdadm /dev/md0 --add /dev/nvme4n1

Mark the failed disk

1
mdadm /dev/md0 --fail /dev/nvme3n1

Remove the failed disk

1
mdadm /dev/md0 --remove /dev/nvme3n1

Check status

1
mdadm --detail /dev/md0

1
2
3
4
5
6
    State : clean, degraded, recovering
    Number   Major   Minor   RaidDevice State
       0     259        0        0      active sync   /dev/nvme0n1
       1     259        1        1      active sync   /dev/nvme1n1
       2     259        4        2      active sync   /dev/nvme2n1
       4     259        2        3      spare rebuilding   /dev/nvme4n1

5.3 Adding a Disk to Expand

Wipe the new disk

1
wipefs -a /dev/nvme5n1

Add a hot spare

1
mdadm /dev/md0 --add /dev/nvme5n1

Adjust the number of RAID devices

1
mdadm /dev/md0 --grow --raid-devices=5

Check status

1
mdadm --detail /dev/md0

1
2
3
4
5
6
7
    State : clean, degraded, reshaping
    Number   Major   Minor   RaidDevice State
       0     259        0        0      active sync   /dev/nvme0n1
       1     259        1        1      active sync   /dev/nvme1n1
       2     259        4        2      active sync   /dev/nvme2n1
       4     259        2        3      spare rebuilding   /dev/nvme4n1
       5     259        5        4      active sync   /dev/nvme5n1

It does not balance data automatically; instead, new data is gradually written to the new disk.

Expand the filesystem

1
xfs_growfs /mnt/mdadm

Check disk size

1
df -h /mnt/mdadm/

1
2
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        2.2T   16G  2.2T   1% /mnt/mdadm

6. Generating Test Data

4M sequential write

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4m_seq_write \
  --rw=write \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

Used to generate test data.

Compute the file’s MD5 checksum

1
md5sum /mnt/mdadm/4m_seq_write.0.0

Used to verify file integrity after adjustments.