This page looks best with JavaScript enabled

ZFS Operations Under RAID

 ·  ☕ 8 min read

ZFS does not support online shrinking; this post focuses on operational tasks such as creating pools, replacing disks, and expanding capacity.

1. ZFS vdev Types

  • mirror: mirroring, multiple disks hold copies of each other, equivalent to RAID 1
  • raidz3: triple parity, tolerates 3 disk failures
  • raidz2: double parity, tolerates 2 disk failures, equivalent to RAID 6
  • raidz1: single parity, tolerates 1 disk failure, equivalent to RAID 5
  • striped: striping, no redundancy, equivalent to RAID 0

They can also be combined:

  • mirror + striped = RAID 10

2. striped (RAID 0)

2.1 Creating a ZFS Pool

  • Install ZFS
1

apt install zfsutils-linux -y
  • List the zfs pools
1

zpool list

View details

1

zpool status tank

If an existing zfs pool already occupies the disks, it must be deleted first.

  • Clean up the zfs pool
1

zpool destroy -f tank
  • Create a striped pool
1
2

zpool create -f -o ashift=12 tank \
  /dev/nvme0n1 /dev/nvme1n1

ashift=12 is the ZFS block size; 12 means 4096 bytes, i.e. 4k. For NVME disks, 12 is recommended.

  • Create a filesystem and mount it
1

zfs create -o mountpoint=/mnt/zfs tank/data
  • Check space
1

df -h /mnt/zfs

1
2

Filesystem      Size  Used Avail Use% Mounted on
tank/data       1.5T  128K  1.5T   1% /mnt/zfs

With two disks, striped stripes across both disks, yielding 100% usable capacity.

2.2 Replacing a Single Disk

  • Wipe the new disk
1

wipefs -a /dev/nvme2n1
  • Replace one disk with another
1

zpool replace tank /dev/nvme1n1 /dev/nvme2n1
  • Check the status
1

zpool status tank

1
2
3

        tank        ONLINE       0     0     0
          nvme0n1   ONLINE       0     0     0
          nvme2n1   ONLINE       0     0     0
  • Check space
1

df -h /mnt/zfs

1
2

Filesystem      Size  Used Avail Use% Mounted on
tank/data       1.5T  128K  1.5T   1% /mnt/zfs

2.3 Adding a Disk to Expand

  • Add a new disk
1

zpool add tank /dev/nvme3n1
  • Check the status
1

zpool status tank

1
2
3
4

        tank        ONLINE       0     0     0
          nvme0n1   ONLINE       0     0     0
          nvme2n1   ONLINE       0     0     0
          nvme3n1   ONLINE       0     0     0
  • Check space
1

df -h /mnt/zfs

1
2

Filesystem      Size  Used Avail Use% Mounted on
tank/data       2.2T  128K  2.2T   1% /mnt/zfs

3. mirror (RAID 1)

3.1 Creating a ZFS Pool

  • Install ZFS
1

apt install zfsutils-linux -y
  • List the zfs pools
1

zpool list

View details

1

zpool status tank

If an existing zfs pool already occupies the disks, it must be deleted first.

  • Clean up the zfs pool
1

zpool destroy -f tank
  • Create a mirror pool
1
2

zpool create -o ashift=12 tank \
  mirror /dev/nvme0n1 /dev/nvme1n1
  • Create a filesystem and mount it
1

zfs create -o mountpoint=/mnt/zfs tank/data
  • Check space
1

df -h /mnt/zfs

1
2

Filesystem      Size  Used Avail Use% Mounted on
tank/data       721G  128K  721G   1% /mnt/zfs

With two disks, mirror mirrors across both disks, yielding 50% usable capacity.

3.2 Replacing a Single Disk

  • Replace one disk with another
1

zpool replace tank /dev/nvme1n1 /dev/nvme2n1
  • Check the status
1

zpool status tank

1
2
3
4

        tank         ONLINE       0     0     0
          mirror-0   ONLINE       0     0     0
            nvme0n1  ONLINE       0     0     0
            nvme2n1  ONLINE       0     0     0
  • Check space
1

df -h /mnt/zfs

1
2

Filesystem      Size  Used Avail Use% Mounted on
tank/data       721G  128K  721G   1% /mnt/zfs

3.3 Adding Disks to Expand

  • Add two disks to make it RAID 10
1

zpool add tank mirror /dev/nvme3n1 /dev/nvme4n1
  • Check the status
1

zpool status tank

1
2
3
4
5
6
7

        tank         ONLINE       0     0     0
          mirror-0   ONLINE       0     0     0
            nvme0n1  ONLINE       0     0     0
            nvme2n1  ONLINE       0     0     0
          mirror-1   ONLINE       0     0     0
            nvme3n1  ONLINE       0     0     0
            nvme4n1  ONLINE       0     0     0
  • Check space
1

df -h /mnt/zfs

1
2

Filesystem      Size  Used Avail Use% Mounted on
tank/data       1.5T  128K  1.5T   1% /mnt/zfs

4. mirror + striped (RAID 10)

4.1 Creating a ZFS Pool

  • Install ZFS
1

apt install zfsutils-linux -y
  • List the zfs pools
1

zpool list

View details

1

zpool status tank

If an existing zfs pool already occupies the disks, it must be deleted first.

  • Clean up the zfs pool
1

zpool destroy -f tank
  • Create a mirror + striped pool
1
2
3

zpool create -f -o ashift=12 tank \
  mirror /dev/nvme0n1 /dev/nvme1n1 \
  mirror /dev/nvme2n1 /dev/nvme3n1

ashift=12 is the ZFS block size; 12 means 4096 bytes, i.e. 4k. For NVME disks, 12 is recommended.

  • Check the pool status
1

zpool status tank

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

  pool: tank
 state: ONLINE
config:

        NAME         STATE     READ WRITE CKSUM
        tank         ONLINE       0     0     0
          mirror-0   ONLINE       0     0     0
            nvme0n1  ONLINE       0     0     0
            nvme1n1  ONLINE       0     0     0
          mirror-1   ONLINE       0     0     0
            nvme2n1  ONLINE       0     0     0
            nvme3n1  ONLINE       0     0     0
  • Create a filesystem and mount it
1

zfs create -o mountpoint=/mnt/zfs tank/data
  • Check space
1

df -h /mnt/zfs

1
2

Filesystem      Size  Used Avail Use% Mounted on
tank/data       1.5T  128K  1.5T   1% /mnt/zfs

With 4 disks, mirror + striped uses two disks for mirroring and two for striping, yielding 50% usable capacity.

4.2 Replacing a Single Disk

  • Replace the old disk
1

zpool replace tank /dev/nvme3n1 /dev/nvme4n1
  • Check the resilvering progress
1

zpool status tank

1
2
3
4
5
6
7
8
9

        tank             ONLINE       0     0     0
          mirror-0       ONLINE       0     0     0
            nvme0n1      ONLINE       0     0     0
            nvme1n1      ONLINE       0     0     0
          mirror-1       ONLINE       0     0     0
            nvme2n1      ONLINE       0     0     0
            replacing-1  ONLINE       0     0     0
              nvme3n1    ONLINE       0     0     0
              nvme4n1    ONLINE       0     0     0  (resilvering)

During resilvering, the filesystem remains fully usable. Resilvering is very fast.

  • Expand the pool if the new disk is larger than the old one
1

zpool online -e tank /dev/nvme4n1

4.3 Adding Disks to Expand

Adding another mirror group, but the data is not redistributed automatically; new writes gradually make use of the new vdev.

  • Add two new disks
1

zpool add tank mirror /dev/nvme3n1 /dev/nvme5n1
  • Check the status
1

zpool status tank

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

        tank         ONLINE       0     0     0
          mirror-0   ONLINE       0     0     0
            nvme0n1  ONLINE       0     0     0
            nvme1n1  ONLINE       0     0     0
          mirror-1   ONLINE       0     0     0
            nvme2n1  ONLINE       0     0     0
            nvme4n1  ONLINE       0     0     0
          mirror-2   ONLINE       0     0     0
            nvme3n1  ONLINE       0     0     0
            nvme5n1  ONLINE       0     0     0
  • Check space
1

zfs list tank/data

1
2

NAME        USED  AVAIL     REFER  MOUNTPOINT
tank/data  50.0G  2.06T     50.0G  /mnt/zfs

1

df -h /mnt/zfs

1
2

Filesystem      Size  Used Avail Use% Mounted on
tank/data       2.2T   51G  2.1T   3% /mnt/zfs

5. raidz2 (RAID 6)

5.1 Creating a ZFS Pool

  • Install ZFS
1

apt install zfsutils-linux -y
  • List the zfs pools
1

zpool list
  • Clean up the zfs pool

If a zfs pool occupies the disks, it must be deleted first.

1

zpool destroy -f tank
  • Create a mirror pool

raidz2 requires at least 4 disks, of which 2 hold data and 2 hold parity.

1
2
3

zpool create -o ashift=12 tank \
  raidz2 /dev/nvme0n1 /dev/nvme1n1 \
         /dev/nvme2n1 /dev/nvme3n1
  • Create a filesystem and mount it
1

zfs create -o mountpoint=/mnt/zfs tank/data
  • Check space
1

df -h /mnt/zfs

1
2

Filesystem      Size  Used Avail Use% Mounted on
tank/data       1.4T  256K  1.4T   1% /mnt/zfs

With 4 disks, raidz2 uses two disks for data and two for parity, yielding 50% usable capacity.

5.2 Replacing a Single Disk

  • Replace one disk with another
1

zpool replace tank /dev/nvme3n1 /dev/nvme4n1
  • Check the status
1

zpool status tank

1
2
3
4
5
6

        tank         ONLINE       0     0     0
          raidz2-0   ONLINE       0     0     0
            nvme0n1  ONLINE       0     0     0
            nvme1n1  ONLINE       0     0     0
            nvme2n1  ONLINE       0     0     0
            nvme4n1  ONLINE       0     0     0
  • Check space
1

df -h /mnt/zfs

1
2

Filesystem      Size  Used Avail Use% Mounted on
tank/data       1.4T  256K  1.4T   1% /mnt/zfs

5.3 Adding Disks to Expand

When expanding raidz2, at least 3 disks are required.

  • Add three new disks
1

zpool add tank raidz2 /dev/nvme3n1 /dev/nvme5n1 /dev/nvme6n1
  • Check the status
1

zpool status tank
  • Check space
1

df -h /mnt/zfs

6. Offlining and Onlining Disks

  • Offline a disk
1

zpool offline tank /dev/nvme0n1
  • Check the status
1

zpool status tank

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

        tank         DEGRADED     0     0     0
          mirror-0   DEGRADED     0     0     0
            nvme0n1  OFFLINE      0     0     0
            nvme1n1  ONLINE       0     0     0
          mirror-1   ONLINE       0     0     0
            nvme2n1  ONLINE       0     0     0
            nvme4n1  ONLINE       0     0     0
          mirror-2   ONLINE       0     0     0
            nvme3n1  ONLINE       0     0     0
            nvme5n1  ONLINE       0     0     0
  • Bring it back online after maintenance
1

zpool online tank /dev/nvme0n1
  • Check the status
1

zpool status tank

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

        tank         ONLINE       0     0     0
          mirror-0   ONLINE       0     0     0
            nvme0n1  ONLINE       0     0     0
            nvme1n1  ONLINE       0     0     0
          mirror-1   ONLINE       0     0     0
            nvme2n1  ONLINE       0     0     0
            nvme4n1  ONLINE       0     0     0
          mirror-2   ONLINE       0     0     0
            nvme3n1  ONLINE       0     0     0
            nvme5n1  ONLINE       0     0     0

7. Backup and Recovery

7.1 Snapshots and Recovery

  • Create a snapshot
1

zfs snapshot tank/data@my_snapshot

A snapshot backs up metadata only and is very fast; the data itself is not copied, and data covered by a snapshot does not free space even when deleted.

  • Restore a snapshot
1

zfs rollback tank/data@my_snapshot

The current filesystem is restored to the state at the time of the snapshot.

  • Restore to another zfs pool
1

zfs send tank/data@my_snapshot | zfs receive backup/data

You need to prepare a backup zfs storage pool in advance; the command above transfers the complete user data to the backup pool.

  • Delete a snapshot
1

zfs destroy tank/data@my_snapshot

7.2 Data Verification

  • Verify the pool data
1

zpool scrub tank
  • Check the scrub progress
1

zpool status tank

8. Generating Test Data

  • 4M sequential writes
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

fio \
  --name=4m_seq_write \
  --rw=write \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/zfs

Used to generate test data.

  • Compute the file’s MD5
1

md5sum /mnt/zfs/4m_seq_write.0.0

Used to verify file integrity after the adjustments.

微信公众号
WRITTEN BY
微信公众号

See Also