Btrfs Operations Under RAID

Btrfs supports online expansion, shrinking, disk replacement, and RAID mode conversion; this post covers creating a filesystem under RAID 0/RAID 1/RAID 10/RAID 6 topologies, adding disks, replacing disks, shrinking, and common operational tasks.

1. Btrfs Profiles

single: single copy, no redundancy, works with one disk or can aggregate multiple disks.
raid0: striping, no redundancy, equivalent to RAID 0.
raid1: mirroring, at least 2 disks, equivalent to RAID 1.
raid10: stripe + mirror, at least 4 disks, equivalent to RAID 10.
raid5: single parity, at least 3 disks, tolerates 1 disk failure, equivalent to RAID 5.
raid6: double parity, at least 4 disks, tolerates 2 disk failures, equivalent to RAID 6.

Data and metadata can be given separate profiles (for example -d raid10 -m raid1); metadata generally uses higher redundancy.

Common concepts:

degraded: a device is missing but the filesystem is still readable (and may even be writable, depending on the profile).
balance: redistribute data chunks across devices to satisfy a new profile or new device; adding disks or changing the RAID level often requires a balance.
replace: replace an old disk with a new one online, with no unmount; the data is copied to the new disk and the old disk is then removed.

2. RAID 0

2.1 Creating a Btrfs Filesystem

Create RAID 0 using multiple disks (both data and metadata are raid0)

1
mkfs.btrfs -d raid0 -m raid0 -f /dev/nvme0n1 /dev/nvme1n1

Mount

You only need to specify one disk; the others will be recognized.

1
2
mkdir -p /mnt/btrfs
mount /dev/nvme0n1 /mnt/btrfs

Check device usage

1
btrfs device usage /mnt/btrfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
/dev/nvme0n1, ID: 1
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 0/2:            1.00GiB
   Metadata,RAID 0/2:      512.00MiB
   System,RAID 0/2:          8.00MiB
   Unallocated:           743.70GiB

/dev/nvme1n1, ID: 2
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 0/2:            1.00GiB
   Metadata,RAID 0/2:      512.00MiB
   System,RAID 0/2:          8.00MiB
   Unallocated:           743.70GiB

Check filesystem usage

1
btrfs filesystem df /mnt/btrfs

1
2
3
4
Data, RAID 0: total=2.00GiB, used=0.00B
System, RAID 0: total=16.00MiB, used=16.00KiB
Metadata, RAID 0: total=1.00GiB, used=208.00KiB
GlobalReserve, single: total=3.25MiB, used=0.00B

Check space

1
df -h /mnt/btrfs

1
2
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1    1.5T  3.5M  1.5T   1% /mnt/btrfs

With two disks, raid0 stripes and yields 100% usable capacity.

2.2 Adding a Disk to Expand

Add a new disk to the mounted filesystem

1
btrfs device add -f /dev/nvme2n1 /mnt/btrfs

-f force-overwrites any existing filesystem.

Check device usage

1
btrfs device usage /mnt/btrfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
/dev/nvme0n1, ID: 1
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 0/2:           25.00GiB
   Metadata,RAID 0/2:      512.00MiB
   System,RAID 0/2:          8.00MiB
   Unallocated:           719.70GiB

/dev/nvme1n1, ID: 2
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 0/2:           25.00GiB
   Metadata,RAID 0/2:      512.00MiB
   System,RAID 0/2:          8.00MiB
   Unallocated:           719.70GiB

/dev/nvme2n1, ID: 3
   Device size:           745.21GiB
   Device slack:              0.00B
   Unallocated:           745.21GiB

Run a balance so the data is distributed across all devices as raid0

1
btrfs balance start -dconvert=raid0 -mconvert=raid0 /mnt/btrfs

The command hangs and you need to wait for the balance to finish. You can open another terminal to check progress.

Check balance progress

1
btrfs balance status /mnt/btrfs

1
10 out of about 27 chunks balanced (11 considered),  63% left

Check device usage

1
btrfs device usage /mnt/btrfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
/dev/nvme0n1, ID: 1
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 0/3:           17.00GiB
   Metadata,RAID 0/3:      352.00MiB
   System,RAID 0/3:         32.00MiB
   Unallocated:           727.84GiB

/dev/nvme1n1, ID: 2
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 0/3:           17.00GiB
   Metadata,RAID 0/3:      352.00MiB
   System,RAID 0/3:         32.00MiB
   Unallocated:           727.84GiB

/dev/nvme2n1, ID: 3
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 0/3:           17.00GiB
   Metadata,RAID 0/3:      352.00MiB
   System,RAID 0/3:         32.00MiB
   Unallocated:           727.84GiB

2.3 Removing a Disk to Shrink

raid0 has no redundancy, so shrinking requires that, at the target disk count, the remaining disks have enough free space to hold the data from the removed disk.

Remove one disk

The data is migrated to the remaining disks automatically.

1
btrfs device remove /dev/nvme2n1 /mnt/btrfs

Check the device list

1
btrfs device usage /mnt/btrfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
/dev/nvme0n1, ID: 1
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 0/2:           25.00GiB
   Metadata,RAID 0/2:      512.00MiB
   System,RAID 0/2:          8.00MiB
   Unallocated:           719.70GiB

/dev/nvme1n1, ID: 2
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 0/2:           25.00GiB
   Metadata,RAID 0/2:      512.00MiB
   System,RAID 0/2:          8.00MiB
   Unallocated:           719.70GiB

3. RAID 1

3.1 Creating a Btrfs Filesystem

Remove the existing filesystem

1
umount /mnt/btrfs

Create RAID 1 using 2 disks

1
mkfs.btrfs -d raid1 -m raid1 -f /dev/nvme0n1 /dev/nvme1n1

Mount

1
2
mkdir -p /mnt/btrfs
mount /dev/nvme0n1 /mnt/btrfs

Check device usage

1
btrfs device usage /mnt/btrfs

Check space

1
df -h /mnt/btrfs

1
2
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1    746G  3.5M  745G   1% /mnt/btrfs

With two disks, raid1 mirrors and yields 50% usable capacity.

3.2 Replacing a Single Disk

Wipe the new disk

1
wipefs -a /dev/nvme2n1

Replace the old disk with the new one

1
btrfs replace start /dev/nvme1n1 /dev/nvme2n1 /mnt/btrfs

Check replacement progress

1
btrfs replace status /mnt/btrfs

1
6.7% done, 0 write errs, 0 uncorr. read errs

The filesystem remains readable and writable during the replacement. Once it completes, the old disk is removed from the filesystem.

If the new disk is larger than the old one, the new space is automatically brought into the available capacity

1
btrfs device usage /mnt/btrfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
/dev/nvme0n1, ID: 1
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 1:             51.00GiB
   Metadata,RAID 1:          1.00GiB
   System,RAID 1:           32.00MiB
   Unallocated:           693.18GiB

/dev/nvme2n1, ID: 2
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 1:             51.00GiB
   Metadata,RAID 1:          1.00GiB
   System,RAID 1:           32.00MiB
   Unallocated:           693.18GiB

3.3 Adding Disks to Expand

Wipe the new disks

1
2
wipefs -a /dev/nvme3n1
wipefs -a /dev/nvme4n1

After adding two disks you can convert to raid10

1
btrfs device add /dev/nvme3n1 /dev/nvme4n1 /mnt/btrfs

Check device usage

1
btrfs device usage /mnt/btrfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
/dev/nvme0n1, ID: 1
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 1:             51.00GiB
   Metadata,RAID 1:          1.00GiB
   System,RAID 1:           32.00MiB
   Unallocated:           693.18GiB

/dev/nvme2n1, ID: 2
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 1:             51.00GiB
   Metadata,RAID 1:          1.00GiB
   System,RAID 1:           32.00MiB
   Unallocated:           693.18GiB

/dev/nvme3n1, ID: 3
   Device size:           745.21GiB
   Device slack:              0.00B
   Unallocated:           745.21GiB

/dev/nvme4n1, ID: 4
   Device size:           745.21GiB
   Device slack:              0.00B
   Unallocated:           745.21GiB

Then balance to raid10

1
btrfs balance start -dconvert=raid10 -mconvert=raid10 /mnt/btrfs

The command hangs and you need to wait for the balance to finish. You can open another terminal to check progress.

Check balance progress

1
btrfs balance status /mnt/btrfs

1
34 out of about 53 chunks balanced (35 considered),  36% left

Check device usage

1
btrfs device usage /mnt/btrfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
/dev/nvme0n1, ID: 1
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 10/4:          51.00GiB
   Metadata,RAID 10/4:       1.00GiB
   System,RAID 10/4:        32.00MiB
   Unallocated:           693.18GiB

/dev/nvme2n1, ID: 2
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 10/4:          51.00GiB
   Metadata,RAID 10/4:       1.00GiB
   System,RAID 10/4:        32.00MiB
   Unallocated:           693.18GiB

/dev/nvme3n1, ID: 3
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 10/4:          51.00GiB
   Metadata,RAID 10/4:       1.00GiB
   System,RAID 10/4:        32.00MiB
   Unallocated:           693.18GiB

/dev/nvme4n1, ID: 4
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 10/4:          51.00GiB
   Metadata,RAID 10/4:       1.00GiB
   System,RAID 10/4:        32.00MiB
   Unallocated:           693.18GiB

3.4 Removing a Disk to Shrink

Remove one disk

1
btrfs device remove /dev/nvme4n1 /mnt/btrfs

Check the devices

1
btrfs device usage /mnt/btrfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
/dev/nvme0n1, ID: 1
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 10/2:          34.00GiB
   Metadata,RAID 10/2:       1.00GiB
   System,RAID 10/2:        32.00MiB
   Unallocated:           710.18GiB

/dev/nvme2n1, ID: 2
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 10/2:          34.00GiB
   Metadata,RAID 10/2:       1.00GiB
   Unallocated:           710.21GiB

/dev/nvme3n1, ID: 3
   Device size:           745.21GiB
   Device slack:              0.00B
   Data,RAID 10/2:          34.00GiB
   System,RAID 10/2:        32.00MiB
   Unallocated:           711.18GiB

4. RAID 10

4.1 Creating a Btrfs Filesystem

Remove the existing filesystem

1
umount /mnt/btrfs

Create RAID 10 using 4 disks

1
mkfs.btrfs -d raid10 -m raid10 -f /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1

-f force-overwrites any existing filesystem.

Mount

1
2
mkdir -p /mnt/btrfs
mount /dev/nvme0n1 /mnt/btrfs

Check device usage

1
btrfs device usage /mnt/btrfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
/dev/nvme0n1, ID: 1
   Data,RAID 10/4:           1.00GiB
   Metadata,RAID 10/4:     256.00MiB

/dev/nvme1n1, ID: 2
   Data,RAID 10/4:           1.00GiB
   Metadata,RAID 10/4:     256.00MiB

/dev/nvme2n1, ID: 3
   Data,RAID 10/4:           1.00GiB
   Metadata,RAID 10/4:     256.00MiB

/dev/nvme3n1, ID: 4
   Data,RAID 10/4:           1.00GiB
   Metadata,RAID 10/4:     256.00MiB

Check space

1
df -h /mnt/btrfs

1
2
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1    1.5T  3.5M  1.5T   1% /mnt/btrfs

With 4 disks, raid10 is stripe + mirror and yields 50% usable capacity.

4.2 Replacing a Single Disk

Wipe the old disk

1
wipefs -a /dev/nvme4n1

Replace the old disk with the new one

1
btrfs replace start /dev/nvme3n1 /dev/nvme4n1 /mnt/btrfs

Check replacement progress

1
btrfs replace status /mnt/btrfs

The filesystem remains usable during the replacement. When it completes, the old disk is removed automatically, and if the new disk is larger the space expands automatically.

Check device usage

1
btrfs device usage /mnt/btrfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
/dev/nvme0n1, ID: 1
   Data,RAID 10/4:          27.00GiB
   Metadata,RAID 10/4:     256.00MiB

/dev/nvme1n1, ID: 2
   Data,RAID 10/4:          27.00GiB
   Metadata,RAID 10/4:     256.00MiB

/dev/nvme2n1, ID: 3
   Data,RAID 10/4:          27.00GiB
   Metadata,RAID 10/4:     256.00MiB

/dev/nvme4n1, ID: 4
   Data,RAID 10/4:          27.00GiB
   Metadata,RAID 10/4:     256.00MiB

4.3 Adding a Disk to Expand

Wipe the new disk

1
wipefs -a /dev/nvme5n1

Add the new disk to the filesystem

1
btrfs device add /dev/nvme5n1 /mnt/btrfs

Balance the data onto the new disk

1
btrfs balance start -dconvert=raid10 -mconvert=raid10 /mnt/btrfs

Check balance progress

1
btrfs balance status /mnt/btrfs

Check device usage

1
btrfs device usage /mnt/btrfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
/dev/nvme0n1, ID: 1
   Data,RAID 10/4:          21.00GiB
   Metadata,RAID 10/4:     512.00MiB

/dev/nvme1n1, ID: 2
   Data,RAID 10/4:          20.00GiB
   Metadata,RAID 10/4:     512.00MiB

/dev/nvme2n1, ID: 3
   Data,RAID 10/4:          20.00GiB
   Metadata,RAID 10/4:     512.00MiB

/dev/nvme4n1, ID: 4
   Data,RAID 10/4:          21.00GiB
   Metadata,RAID 10/4:     512.00MiB

/dev/nvme5n1, ID: 5
   Data,RAID 10/4:          22.00GiB
   Metadata,RAID 10/4:     512.00MiB

4.4 Removing a Disk to Shrink

You must ensure that after removing the disk the disk count is not below RAID’s minimum requirement, and that the remaining disks have enough space to hold the data from the removed disk.

Remove one disk

1
btrfs device remove /dev/nvme0n1 /mnt/btrfs

Check the device list

1
btrfs device usage /mnt/btrfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
/dev/nvme1n1, ID: 2
   Data,RAID 10/4:          26.00GiB
   Metadata,RAID 10/4:     512.00MiB

/dev/nvme2n1, ID: 3
   Data,RAID 10/4:          26.00GiB
   Metadata,RAID 10/4:     512.00MiB

/dev/nvme4n1, ID: 4
   Data,RAID 10/4:          26.00GiB
   Metadata,RAID 10/4:     512.00MiB

/dev/nvme5n1, ID: 5
   Data,RAID 10/4:          26.00GiB
   Metadata,RAID 10/4:     512.00MiB

5. RAID 6

5.1 Creating a Btrfs Filesystem

Remove the existing filesystem

1
umount /mnt/btrfs

Create RAID 6

At least 4 disks.

1
mkfs.btrfs -d raid6 -m raid6 -f /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1

Mount

1
2
mkdir -p /mnt/btrfs
mount /dev/nvme0n1 /mnt/btrfs

Check devices and profiles

1
btrfs device usage /mnt/btrfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
/dev/nvme0n1, ID: 1
   Data,RAID 6/4:            1.00GiB
   Metadata,RAID 6/4:      256.00MiB

/dev/nvme1n1, ID: 2
   Data,RAID 6/4:            1.00GiB
   Metadata,RAID 6/4:      256.00MiB

/dev/nvme2n1, ID: 3
   Data,RAID 6/4:            1.00GiB
   Metadata,RAID 6/4:      256.00MiB

/dev/nvme3n1, ID: 4
   Data,RAID 6/4:            1.00GiB
   Metadata,RAID 6/4:      256.00MiB

Check space

1
df -h /mnt/btrfs

1
/dev/nvme0n1    3.0T  3.5M  1.5T   1% /mnt/btrfs

RAID 6 with 4 disks yields 50% usable capacity (2 data + 2 parity).

5.2 Replacing a Single Disk

Wipe the old disk

1
wipefs -a /dev/nvme4n1

Replace the old disk with the new one

1
btrfs replace start /dev/nvme3n1 /dev/nvme4n1 /mnt/btrfs

Check progress

1
btrfs replace status /mnt/btrfs

Wait for it to finish.

Check device usage

1
btrfs device usage /mnt/btrfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
/dev/nvme0n1, ID: 1
   Data,RAID 6/4:           26.00GiB
   Metadata,RAID 6/4:      256.00MiB

/dev/nvme1n1, ID: 2
   Data,RAID 6/4:           26.00GiB
   Metadata,RAID 6/4:      256.00MiB

/dev/nvme2n1, ID: 3
   Data,RAID 6/4:           26.00GiB
   Metadata,RAID 6/4:      256.00MiB

/dev/nvme4n1, ID: 4
   Data,RAID 6/4:           26.00GiB
   Metadata,RAID 6/4:      256.00MiB

5.3 Adding a Disk to Expand

Add a new disk

1
btrfs device add -f /dev/nvme5n1 /mnt/btrfs

Balance the data onto the new disk

1
btrfs balance start -dconvert=raid6 -mconvert=raid6 /mnt/btrfs

1
2
3
4
5
6
7
WARNING:

        RAID 5/6 support has known problems and is strongly discouraged
        to be used besides testing or evaluation. It is recommended that
        you use one of the other RAID profiles.
        The operation will continue in 10 seconds.
        Use Ctrl-C to stop.

A warning appears noting that RAID 5/6 is not recommended for production and that the data is unreliable.

Check device usage

1
btrfs device usage /mnt/btrfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
/dev/nvme0n1, ID: 1
   Data,RAID 6/5:           18.00GiB
   Metadata,RAID 6/5:      352.00MiB

/dev/nvme1n1, ID: 2
   Data,RAID 6/5:           18.00GiB
   Metadata,RAID 6/5:      352.00MiB

/dev/nvme2n1, ID: 3
   Data,RAID 6/5:           18.00GiB
   Metadata,RAID 6/5:      352.00MiB

/dev/nvme4n1, ID: 4
   Data,RAID 6/5:           18.00GiB
   Metadata,RAID 6/5:      352.00MiB

/dev/nvme5n1, ID: 5
   Data,RAID 6/5:           18.00GiB
   Metadata,RAID 6/5:      352.00MiB

5.4 Removing a Disk to Shrink

Remove one disk

1
btrfs device remove /dev/nvme0n1 /mnt/btrfs

When removing a device, no warning appeared.

Check device usage

1
btrfs device usage /mnt/btrfs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
/dev/nvme1n1, ID: 2
   Data,RAID 6/4:           26.00GiB
   Metadata,RAID 6/4:      512.00MiB

/dev/nvme2n1, ID: 3
   Data,RAID 6/4:           26.00GiB
   Metadata,RAID 6/4:      512.00MiB

/dev/nvme4n1, ID: 4
   Data,RAID 6/4:           26.00GiB
   Metadata,RAID 6/4:      512.00MiB

/dev/nvme5n1, ID: 5
   Data,RAID 6/4:           26.00GiB
   Metadata,RAID 6/4:      512.00MiB

6. Degradation and Recovery

When a disk fails or is pulled, the filesystem may be in a degraded state and is still readable (and may even be writable, depending on the profile).
Check the status

1
2
btrfs device usage /mnt/btrfs
btrfs filesystem show /mnt/btrfs

Recovery options:
- Replace a disk: use btrfs replace start old-disk new-disk /mnt/btrfs for an online replacement.
- Remove a disk: if the disk is no longer needed, use btrfs device remove missing-device (either when the “missing” device can be identified, or you can first btrfs device add a new disk and then balance/remove).
During a replace or balance it is best to avoid losing another disk, so that recovery goes faster.

7. Backup and Recovery

7.1 Snapshots and Recovery

Create a snapshot

1
btrfs subvolume snapshot /mnt/btrfs /mnt/btrfs/snap_test

The snapshot created contains only metadata.

View the snapshot files

1
ls /mnt/btrfs/snap_test

Very easy to use — the snapshot is directly accessible as a file directory.

Delete a snapshot

1
btrfs subvolume delete /mnt/btrfs/snap_test

7.2 Data Verification

Verify the filesystem data

1
btrfs scrub start /mnt/btrfs

Check scrub progress

1
btrfs scrub status /mnt/btrfs

1
2
Bytes scrubbed:   7.84GiB  (15.62%)
Rate:             1.57GiB/s

It is advisable to run a scrub periodically to find silent errors (a bit flipped without being noticed).

8. Generating Test Data

4M sequential writes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
fio \
  --name=4m_seq_write \
  --rw=write \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/btrfs

Used to generate test data.

Compute the file’s MD5

1
md5sum /mnt/btrfs/4m_seq_write.0.0

Used to verify file integrity after the adjustments.