# ZFS RAIDZ2 with ZFSBootMenu: A Fault-Tolerant 4-Disk NAS


In my [previous article]({{< relref "Install-ubuntu-on-zfs-mirror-with-ZFSBootMenu.md" >}}), I described how to install Ubuntu on a ZFS mirror and boot it using ZFSBootMenu.

This time, the goal is more ambitious: build a proper four-disk NAS where both the operating system and user data live on a single ZFS RAIDZ2 pool, while still being able to boot the machine after losing one or even two physical disks.

I also wanted to avoid a dedicated boot SSD.

The final design looks like this:

```text
HDD1
├── 1 GiB ESP
└── ZFS ─┐

HDD2     │
├── ESP  │
└── ZFS ─┤
         │
HDD3     ├── rpool / RAIDZ2
├── ESP  │
└── ZFS ─┤
         │
HDD4     │
├── ESP  │
└── ZFS ─┘
```

Each disk has its own EFI System Partition, containing a copy of ZFSBootMenu.

The ZFS pool itself contains Ubuntu and NAS data.

The system was also tested by physically removing disks, including a two-disk failure scenario.

## Hardware and software

My setup is:

| Component | Configuration |
|---|---|
| NAS | TerraMaster F4-425 |
| HDDs | 4 × WD Red WD60EFRX, 6 TB |
| RAM | 16 GB |
| OS | Ubuntu Server 26.04 LTS |
| Kernel | Linux 7.0.x |
| OpenZFS | 2.4.x |
| Pool layout | RAIDZ2 |
| Bootloader | ZFSBootMenu |
| ESPs | One 1 GiB ESP per HDD |

The internal TerraMaster USB DOM is left untouched.

The important design decision is that every HDD is independently bootable.

A disk therefore looks like this:

```text
Disk
├── GPT partition 1: EFI System Partition
└── GPT partition 2: ZFS
```

And each ESP contains:

```text
EFI/ZBM/VMLINUZ.EFI
EFI/BOOT/BOOTX64.EFI
```

The second file is the standard UEFI fallback path.

## Installing Ubuntu on RAIDZ2

### Preparing the live environment

Boot Ubuntu 26.04 installation media in x86_64 UEFI mode. The commands through
the first reboot run in a root Bash shell. Secure Boot must be disabled for the
unsigned upstream EFI image used below; signing and enrolling your own image
is a separate setup.

```bash
sudo -i
test -d /sys/firmware/efi/efivars
apt update
apt install -y debootstrap gdisk dosfstools parted zfsutils-linux curl ca-certificates
test -e /usr/share/debootstrap/scripts/resolute
```

Create the host ID before creating the pool, then copy it into the new system
later. Keep this value stable across boots:

```bash
zgenhostid
hostid
```

### Preparing the disks

First, identify the drives.

Do not rely on `/dev/sda`, `/dev/sdb`, and similar names when creating the ZFS pool.

Use stable `/dev/disk/by-id` paths instead.

For example:

```bash
ls -l /dev/disk/by-id/
lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE,FSTYPE
```

Define the disks:

```bash
D1=/dev/disk/by-id/ata-DISK1
D2=/dev/disk/by-id/ata-DISK2
D3=/dev/disk/by-id/ata-DISK3
D4=/dev/disk/by-id/ata-DISK4

DISKS=(
  "$D1"
  "$D2"
  "$D3"
  "$D4"
)
export D1 D2 D3 D4
```

The following commands destroy the existing partition tables, so double-check the selected disks before continuing.

Create GPT tables and two partitions per disk:

```bash
for d in "${DISKS[@]}"; do
    wipefs -a "$d"
    sgdisk --zap-all "$d"

    sgdisk \
      -n 1:1M:+1G \
      -t 1:EF00 \
      -c 1:EFI \
      "$d"

    sgdisk \
      -n 2:0:0 \
      -t 2:6A85CF4D-1DD2-11B2-99A6-080020736631 \
      -c 2:ZFS \
      "$d"
done

partprobe
udevadm settle
```

Verify:

```bash
lsblk -o NAME,SIZE,PARTTYPE,FSTYPE
```

### Creating the four EFI System Partitions

Format partition 1 on every disk:

```bash
mkfs.vfat -F 32 -n ZBM1 "${D1}-part1"
mkfs.vfat -F 32 -n ZBM2 "${D2}-part1"
mkfs.vfat -F 32 -n ZBM3 "${D3}-part1"
mkfs.vfat -F 32 -n ZBM4 "${D4}-part1"
```

Then get their UUIDs:

```bash
lsblk -o PATH,FSTYPE,LABEL,UUID,PARTTYPE
```

These UUIDs will later be used by `/etc/fstab` and `zbm-esp-sync`.

### Creating the RAIDZ2 pool

The compatibility profile limits enabled pool features to ones supported by
the selected ZFSBootMenu image. Do not run `zpool upgrade` without checking
bootloader support in the [upstream release notes](https://github.com/zbm-dev/zfsbootmenu/releases/tag/v3.1.0).

Create the pool:

```bash
zpool create -f \
    -o ashift=12 \
    -o compatibility=openzfs-2.3-linux \
    -O compression=lz4 \
    -O acltype=posixacl \
    -O xattr=sa \
    -O relatime=on \
    -O mountpoint=none \
    -R /mnt \
    rpool \
    raidz2 \
    "${D1}-part2" \
    "${D2}-part2" \
    "${D3}-part2" \
    "${D4}-part2"
```

I use:

```text
ashift=12
```

for 4 KiB sectors.

Compression is enabled with:

```text
compression=lz4
```

For Linux systems using POSIX ACLs, I use:

```text
acltype=posixacl
xattr=sa
```

Verify the pool:

```bash
zpool status
zpool get ashift,compatibility rpool
```

### Creating the root datasets

Create a container for boot environments:

```bash
zfs create \
    -o canmount=off \
    -o mountpoint=none \
    rpool/ROOT
```

Create the Ubuntu root filesystem:

```bash
zfs create \
    -o canmount=noauto \
    -o mountpoint=/ \
    rpool/ROOT/ubuntu
```

Mount it:

```bash
zfs mount rpool/ROOT/ubuntu
```

Because the pool was imported with:

```text
-R /mnt
```

the root filesystem appears under `/mnt`.

Create a separate `/home` dataset:

```bash
zfs create \
    -o mountpoint=/home \
    rpool/home
```

Set the boot filesystem:

```bash
zpool set bootfs=rpool/ROOT/ubuntu rpool
```

Set the kernel command line inherited by boot environments:

```bash
zfs set \
    org.zfsbootmenu:commandline="quiet" \
    rpool/ROOT
```

There is an important distinction here.

This property:

```text
org.zfsbootmenu:commandline
```

controls the Linux kernel launched by ZFSBootMenu.

Parameters such as:

```text
zbm.import_policy=force
```

control ZFSBootMenu itself and are stored in the command line embedded inside the ZFSBootMenu EFI executable.

### Installing Ubuntu 26.04 with debootstrap

Bootstrap the system:

```bash
debootstrap --arch=amd64 resolute /mnt http://archive.ubuntu.com/ubuntu/
```

Bind the required virtual filesystems:

```bash
mount --rbind /dev  /mnt/dev
mount --make-rslave /mnt/dev

mount --rbind /proc /mnt/proc
mount --make-rslave /mnt/proc

mount --rbind /sys /mnt/sys
mount --make-rslave /mnt/sys

mount --rbind /run /mnt/run
mount --make-rslave /mnt/run
```

Copy DNS configuration:

```bash
cp -L /etc/resolv.conf /mnt/etc/resolv.conf
cp /etc/hostid /mnt/etc/hostid
```

Enter the new system:

```bash
chroot /mnt /bin/bash
```

All following installation commands run **inside the chroot**, until the
explicit `exit` in “First boot”. Configure the Ubuntu repositories, including
Universe and security updates, before installing packages:

```bash
rm -f /etc/apt/sources.list
mkdir -p /etc/apt/sources.list.d
cat > /etc/apt/sources.list.d/ubuntu.sources <<'EOF'
Types: deb
URIs: http://archive.ubuntu.com/ubuntu/
Suites: resolute resolute-updates resolute-backports
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

Types: deb
URIs: http://security.ubuntu.com/ubuntu/
Suites: resolute-security
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
EOF
export LANG=C.UTF-8
export LC_ALL=C.UTF-8
```

Install the required packages:

```bash
apt update

apt install -y \
    linux-generic \
    ubuntu-server-minimal \
    openssh-server \
    sudo \
    zfsutils-linux \
    zfs-initramfs \
    dosfstools \
    efibootmgr \
    curl \
    ca-certificates \
    binutils \
    locales \
    netplan.io
```

### Configuring the host ID

A stable `/etc/hostid` is important for root-on-ZFS systems.

Verify the ID copied from the live environment:

```bash
hostid
```

Verify:

```bash
ls -l /etc/hostid
```

### Hostname, user and SSH

Set the hostname:

```bash
echo nas > /etc/hostname
printf '127.0.1.1\tnas\n' >> /etc/hosts
```

Create an administrative user:

```bash
useradd \
    --create-home \
    --shell /bin/bash \
    --groups sudo \
    nasadmin

passwd nasadmin
```

Enable SSH:

```bash
systemctl enable ssh
```

### Network configuration

Example `/etc/netplan/01-netcfg.yaml`:

```yaml
network:
  version: 2
  renderer: networkd

  ethernets:
    enp1s0:
      dhcp4: true
```

Set appropriate permissions:

```bash
chmod 600 /etc/netplan/01-netcfg.yaml
netplan generate
```

### Creating zpool.cache

There is a small but important detail here.

The pool was created with:

```text
-R /mnt
```

which sets an alternate root and effectively leaves the pool without a persistent cache file.

Create one explicitly:

```bash
mkdir -p /etc/zfs

zpool set \
    cachefile=/etc/zfs/zpool.cache \
    rpool
```

Verify:

```bash
ls -lh /etc/zfs/zpool.cache
```

Enable ZFS services for the installed system. `enable` creates the links needed
at boot; it does not try to start a systemd instance inside the chroot:

```bash
systemctl enable zfs.target zfs-import-cache zfs-mount zfs-import.target
```

Rebuild the initramfs:

```bash
update-initramfs -u -k all
```

Check that both the host ID and pool cache were included:

```bash
lsinitramfs /boot/initrd.img-* \
    | grep -E 'hostid|zpool.cache'
```

You should see:

```text
etc/hostid
etc/zfs/zpool.cache
```

## Installing ZFSBootMenu

### Installing the EFI binary

I use the official prebuilt ZFSBootMenu EFI binary.

Mount the primary ESP at:

```text
/boot/efi
```

For example:

```bash
mkdir -p /boot/efi
mount /dev/disk/by-uuid/AAAA-BBBB /boot/efi
```

Create the directories:

```bash
mkdir -p \
    /boot/efi/EFI/ZBM \
    /boot/efi/EFI/BOOT
```

Download a pinned [ZFSBootMenu 3.1.0 release](https://github.com/zbm-dev/zfsbootmenu/releases/tag/v3.1.0)
and verify its published SHA256 before installing it. This example selects the
x86_64 recovery image with Linux 6.12; its kernel is separate from Ubuntu's:

```bash
ZBM_VERSION=v3.1.0
ZBM_IMAGE="zfsbootmenu-recovery-x86_64-${ZBM_VERSION}-linux6.12.EFI"
ZBM_URL="https://github.com/zbm-dev/zfsbootmenu/releases/download/${ZBM_VERSION}"
ZBM_DOWNLOAD=$(mktemp -d)
cd "$ZBM_DOWNLOAD"
curl --fail --location --remote-name "${ZBM_URL}/${ZBM_IMAGE}"
curl --fail --location --remote-name "${ZBM_URL}/sha256.txt"
grep -F "(${ZBM_IMAGE})" sha256.txt > selected.sha256
sha256sum --check selected.sha256 || exit 1
install -m 0644 "$ZBM_IMAGE" /boot/efi/EFI/ZBM/VMLINUZ.EFI
```

The checksum detects a corrupt download. Upstream also documents signature
verification in [Binary Releases](https://docs.zfsbootmenu.org/en/v3.1.x/general/binary-releases.html).

Install `zbm-kcl` from the source commit tagged `v3.1.0`; `binutils` supplies
the `objcopy` command it uses to edit EFI command lines. The release's
`sha256.txt` covers the EFI images and tarballs, not this standalone script.
The digest below was calculated from the [pinned upstream source](https://github.com/zbm-dev/zfsbootmenu/blob/7174e420590a270ece3c6426bdb1d5cdd9ef27b0/bin/zbm-kcl):

```bash
ZBM_COMMIT=7174e420590a270ece3c6426bdb1d5cdd9ef27b0
curl --fail --location --output zbm-kcl \
    "https://raw.githubusercontent.com/zbm-dev/zfsbootmenu/${ZBM_COMMIT}/bin/zbm-kcl" || exit 1
printf '%s  %s\n' \
    '16edae3eee5df9a0b133734bc4d8a8cb68ca73ac30f98cb21cea3212b051ff01' \
    zbm-kcl | sha256sum --check - || exit 1
install -m 0755 zbm-kcl /usr/local/sbin/zbm-kcl
zbm-kcl /boot/efi/EFI/ZBM/VMLINUZ.EFI
```

Keep an unmodified recovery image and create the standard UEFI fallback copy:

```bash
cp /boot/efi/EFI/ZBM/VMLINUZ.EFI /boot/efi/EFI/ZBM/VMLINUZ-BACKUP.EFI
cp /boot/efi/EFI/ZBM/VMLINUZ.EFI /boot/efi/EFI/BOOT/BOOTX64.EFI
cd /
```

### Mounting only the primary ESP

Only the primary ESP is mounted permanently.

Example `/etc/fstab`:

```fstab
UUID=AAAA-BBBB /boot/efi vfat defaults,nofail,umask=0077,x-systemd.device-timeout=5s 0 0
```

The key option is:

```text
nofail
```

If the disk containing the primary ESP disappears, the operating system must still be able to boot from another ESP.

### Creating UEFI boot entries

Create a UEFI entry for every disk.

Use the stable disk paths exported before entering the chroot:

```bash
i=1
for disk in "$D1" "$D2" "$D3" "$D4"; do
    efibootmgr --create --disk "$disk" --part 1 \
      --label "ZFSBootMenu HDD${i}" --loader '\EFI\ZBM\VMLINUZ.EFI'
    i=$((i + 1))
done
```

The backup ESPs will receive their EFI files in the synchronization step below.

Verify:

```bash
efibootmgr
```

The firmware should have entries similar to:

```text
ZFSBootMenu HDD1
ZFSBootMenu HDD2
ZFSBootMenu HDD3
ZFSBootMenu HDD4
```

In addition, every ESP contains:

```text
EFI/BOOT/BOOTX64.EFI
```

so the firmware can also use the generic UEFI fallback path.

### Synchronizing ESPs with zbm-esp-sync

ZFS protects the contents of the pool.

It does not replicate FAT32 ESPs.

For that purpose I use a small tool I wrote called [`zbm-esp-sync`](https://gitlab.com/tty8747/zbm-esp-sync).

The design is intentionally simple:

```text
ESP HDD1 = master
       │
       ├── ESP HDD2
       ├── ESP HDD3
       └── ESP HDD4
```

Only these trees are synchronized:

```text
EFI/ZBM
EFI/BOOT
```

The utility verifies that the target really is an EFI System Partition and that the filesystem is VFAT.

It does not modify partition tables or format drives.

Install the build dependencies:

```bash
apt install -y \
    golang-go \
    git \
    make \
    util-linux
```

Clone and build:

```bash
git clone --branch v0.1.1 --depth 1 \
    https://gitlab.com/tty8747/zbm-esp-sync \
    /usr/local/src/zbm-esp-sync

cd /usr/local/src/zbm-esp-sync

go test ./...
make VERSION=v0.1.1 build
make VERSION=v0.1.1 install
zbm-esp-sync --version
cd /
```

The upstream install target places the binary in `/usr/local/sbin` and installs
the systemd units. Replace all example UUIDs in `/etc/zbm-esp-sync/config.yaml`:

```yaml
master: /dev/disk/by-uuid/AAAA-BBBB

esp:
  - /dev/disk/by-uuid/AAAA-BBBB
  - /dev/disk/by-uuid/CCCC-DDDD
  - /dev/disk/by-uuid/EEEE-FFFF
  - /dev/disk/by-uuid/GGGG-HHHH

paths:
  - EFI/ZBM
  - EFI/BOOT
```

Test first:

```bash
sudo zbm-esp-sync refresh --dry-run
```

Then synchronize:

```bash
sudo zbm-esp-sync refresh
sudo zbm-esp-sync verify
sudo zbm-esp-sync status
```

A healthy setup should look roughly like:

```text
ESP                          ROLE    MOUNTED  FILES  CHECKSUM  STATE
/dev/disk/by-uuid/...        master  yes      ok     ok        current
/dev/disk/by-uuid/...        backup  no       ok     ok        current
/dev/disk/by-uuid/...        backup  no       ok     ok        current
/dev/disk/by-uuid/...        backup  no       ok     ok        current
```

Enable automatic synchronization for the next boot. Do not start services with
`--now` inside the chroot:

```bash
systemctl enable zbm-esp-sync.path
```

The path unit watches the master ESP for changes to:

```text
EFI/ZBM
EFI/BOOT
```

When files change, synchronization is triggered automatically.

### What happens when the primary ESP disappears

If HDD1 is physically missing, the command:

```bash
sudo zbm-esp-sync status
```

may report something similar to:

```text
master ... not a block device
```

This is expected.

`zbm-esp-sync` is not involved in booting Linux.

The NAS can continue running on the remaining disks, but I prefer not to update ZFSBootMenu while the master ESP is absent.

### The degraded-pool boot problem

The first physical failure test exposed an interesting issue.

With HDD1 removed, ZFSBootMenu itself started successfully from another disk, but the ZFS pool was not automatically imported.

The screen repeatedly showed:

```text
Unable to import pool
```

From the ZFSBootMenu recovery shell, this worked:

```bash
zpool import -f -N rpool
```

After that, Ubuntu booted normally.

That tells us that the redundancy of the RAIDZ2 pool itself is fine, but the import policy is blocking unattended degraded boot.

For this standalone NAS I changed the ZFSBootMenu import policy to:

```text
zbm.import_policy=force
```

> **Warning**
>
> `zbm.import_policy=force` makes ZFSBootMenu use `zpool import -f`.
> Enable it only when the pool cannot be active on another host; importing
> the same pool on two hosts can corrupt data. See the
> [ZFSBootMenu import policy documentation](https://docs.zfsbootmenu.org/en/v3.1.x/man/zfsbootmenu.7.html#command-line-parameters).

### Changing the ZFSBootMenu import policy

For a prebuilt ZFSBootMenu EFI binary, the command line can be edited using `zbm-kcl`.

Inspect the current value:

```bash
sudo zbm-kcl \
    /boot/efi/EFI/ZBM/VMLINUZ.EFI
```

Change the import policy:

```bash
sudo zbm-kcl \
    -r zbm.import_policy \
    -a zbm.import_policy=force \
    /boot/efi/EFI/ZBM/VMLINUZ.EFI
```

Verify:

```bash
sudo zbm-kcl \
    /boot/efi/EFI/ZBM/VMLINUZ.EFI
```

In my setup the result is:

```text
quiet loglevel=0 nomodeset zbm.import_policy=force
```

Update the generic fallback copy:

```bash
sudo cp \
    /boot/efi/EFI/ZBM/VMLINUZ.EFI \
    /boot/efi/EFI/BOOT/BOOTX64.EFI
```

Then synchronize all ESPs:

```bash
sudo zbm-esp-sync refresh
sudo zbm-esp-sync verify
```

I intentionally keep:

```text
VMLINUZ-BACKUP.EFI
```

with the normal hostid-based import policy.

That gives me an emergency ZFSBootMenu image that does not force-import pools.

### First boot

Before leaving the chroot, check the root dataset, boot files and ESP copies:

```bash
zpool get bootfs rpool
zfs list
ls -l /boot/vmlinuz-* /boot/initrd.img-*
zbm-esp-sync verify
efibootmgr
```

Exit to the live shell, unmount the installation tree and export the pool. If
unmount or export fails, stop and resolve the busy mount before rebooting:

```bash
exit
cd /
umount --recursive /mnt
zpool export rpool
```

After both commands succeed, reboot and remove the installation medium:

```bash
reboot
```

Log in as `nasadmin` on the installed system. Commands from this point run on
the NAS, with `sudo` where required:

```bash
findmnt /
zpool status
hostid
sudo systemctl start zbm-esp-sync.service
sudo systemctl enable --now zbm-esp-sync.path
systemctl status zbm-esp-sync.path
sudo journalctl -u zbm-esp-sync.service --no-pager -n 30
sudo zbm-esp-sync verify
```

Confirm normal boot with all disks present before testing missing disks.

## Testing fault tolerance

RAIDZ2 theory is useful, but boot redundancy should be tested physically.

Always shut the NAS down before removing drives:

```bash
sudo poweroff
```

### One-disk failure

First I removed HDD1.

The system successfully booted from another ESP.

The pool became:

```text
rpool       DEGRADED
  raidz2-0  DEGRADED
    HDD1    UNAVAIL
    HDD2    ONLINE
    HDD3    ONLINE
    HDD4    ONLINE
```

The filesystems were still mounted normally:

```text
/      rpool/ROOT/ubuntu
/home  rpool/home
```

SSH came up without manual intervention.

After reinstalling HDD1, ZFS performed a very small resilver and the pool returned to:

```text
ONLINE
```

### Two-disk failure

The more important test was removing HDD1 and HDD2 at the same time.

Only HDD3 and HDD4 remained.

The boot sequence was:

```text
UEFI
 ↓
ESP on HDD3
 ↓
ZFSBootMenu
 ↓
zbm.import_policy=force
 ↓
rpool DEGRADED
 ↓
Ubuntu
 ↓
SSH
```

The pool looked like this:

```text
rpool       DEGRADED
  raidz2-0  DEGRADED
    HDD1    UNAVAIL
    HDD2    UNAVAIL
    HDD3    ONLINE
    HDD4    ONLINE

errors: No known data errors
```

This confirmed not only that RAIDZ2 can operate with two missing drives, but also that the entire boot path remains functional.

That includes:

```text
firmware
ESP
ZFSBootMenu
pool import
root-on-ZFS
systemd
network
SSH
```

## ZFS tuning for a NAS

### Will zbm-esp-sync wear out the ESPs?

No.

There are two separate questions here.

First, these ESPs live on mechanical HDDs, not NAND flash.

There is no SSD-style TBW endurance concern.

Second, `zbm-esp-sync` does not rewrite the partitions continuously.

Normal operation looks like:

```text
ZFSBootMenu unchanged
        ↓
nothing is written

ZFSBootMenu updated
        ↓
master ESP changes
        ↓
systemd.path triggers
        ↓
one synchronization run
        ↓
backup ESPs are updated
```

The tool compares content and only updates changed files.

It writes temporary files, renames them into place and verifies the resulting data.

It never modifies GPT or formats filesystems.

A more relevant HDD wear issue is excessive power cycling or aggressive spindown.

For mechanical disks, I would pay much more attention to:

```text
Start_Stop_Count
Load_Cycle_Count
Power_Cycle_Count
Power_On_Hours
Temperature
```

than to a few dozen megabytes written to an ESP during a bootloader update.

### Should I create a RAM disk for ZFS cache?

No.

ZFS already has a native RAM cache called ARC.

On this NAS with 16 GB of RAM, the system reported approximately:

```text
Total RAM:      15.4 GiB
ARC max:        14.4 GiB
ARC current:    ~143 MiB
```

The ARC was still tiny because the NAS had just been installed and there was almost no workload.

There is no benefit in doing this:

```text
RAM
 ↓
tmpfs
 ↓
L2ARC
 ↓
ZFS
```

ZFS already does:

```text
RAM
 ├── ARC
 │    └── read cache
 │
 └── dirty data
      └── write buffering
             ↓
            ZFS
```

L2ARC is meant to be a second-level read cache, usually on SSD or NVMe, when the working set no longer fits into RAM.

It is not a replacement for ARC.

Likewise, a SLOG is not a generic write cache.

For now I leave ARC tuning at its defaults:

```text
zfs_arc_max=0
zfs_arc_min=0
```

This allows OpenZFS to manage memory dynamically.

Useful commands:

```bash
zarcsummary
```

and:

```bash
zarcstat 5
```

### Dataset layout for NAS data

I prefer not to store all user data directly inside one generic dataset.

Separate datasets make it possible to apply different snapshot policies, quotas and filesystem properties.

For example:

```bash
sudo zfs create \
    -o mountpoint=/srv/data \
    rpool/data

sudo zfs create \
    -o mountpoint=/srv/data/shared \
    -o atime=off \
    rpool/data/shared

sudo zfs create \
    -o mountpoint=/srv/data/documents \
    -o atime=off \
    rpool/data/documents

sudo zfs create \
    -o mountpoint=/srv/data/media \
    -o atime=off \
    rpool/data/media

sudo zfs create \
    -o mountpoint=/srv/data/backups \
    -o atime=off \
    rpool/data/backups
```

The result is:

```text
rpool
├── ROOT
│   └── ubuntu
├── home
└── data
    ├── shared
    ├── documents
    ├── media
    └── backups
```

For Samba-oriented datasets I use:

```text
acltype=posixacl
xattr=sa
compression=lz4
```

Disabling `atime` is useful for datasets where applications do not care about file access timestamps.

I would not blindly set a large `recordsize` everywhere.

`recordsize` should be tuned per workload.

Large sequential media or backup files may benefit from a larger value, but mixed file workloads are often better left at the default.

## Data protection

### ZFS snapshots

RAIDZ2 protects against physical disk failure.

It does not protect against:

```text
rm -rf
accidental overwrite
ransomware
broken application logic
operator mistakes
```

Snapshots help with these problems.

A manual snapshot:

```bash
SNAPSHOT="manual-$(date -u +%Y%m%dT%H%M%SZ)"
sudo zfs snapshot -r "rpool/data@${SNAPSHOT}"
```

The `-r` option includes `shared`, `documents`, `media` and `backups`. Without
it, only the parent dataset is snapshotted. Verify all child snapshots:

```bash
zfs list -t snapshot -r rpool/data
```

Snapshots are very cheap to create because they are initially just references to existing blocks.

However, a snapshot is not a backup.

If the entire pool disappears, the snapshots disappear with it.

### Automatic snapshots with Sanoid

For automated retention policies, I use Sanoid.

Install it:

```bash
sudo apt install sanoid
```

Example `/etc/sanoid/sanoid.conf`:

```ini
[rpool/data]
    use_template = nas
    recursive = yes

[rpool/ROOT/ubuntu]
    use_template = system

[template_nas]
    frequently = 0
    hourly = 24
    daily = 30
    weekly = 8
    monthly = 12
    yearly = 0
    autosnap = yes
    autoprune = yes

[template_system]
    frequently = 0
    hourly = 0
    daily = 7
    weekly = 4
    monthly = 3
    yearly = 0
    autosnap = yes
    autoprune = yes
```

This keeps approximately:

```text
NAS data:
24 hourly
30 daily
8 weekly
12 monthly

Ubuntu root:
7 daily
4 weekly
3 monthly
```

Enable the timer:

```bash
sudo systemctl enable --now sanoid.timer
```

Check it:

```bash
systemctl status sanoid.timer
sudo sanoid --monitor-snapshots
```

For off-host backup, Sanoid is commonly paired with Syncoid and ZFS replication.

### Regular ZFS scrub

RAIDZ2 provides redundancy against disk failures. Regular scrubs complement
that protection by detecting latent corruption: a scrub reads allocated data,
verifies checksums and repairs damaged blocks from redundancy when possible.
See the [OpenZFS scrub documentation](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-scrub.8.html).

Start a manual scrub:

```bash
sudo zpool scrub rpool
```

Wait until it finishes:

```bash
sudo zpool scrub -w rpool
```

Then check:

```bash
sudo zpool status -v
```

A healthy result looks like:

```text
scan: scrub repaired 0B ... with 0 errors
errors: No known data errors
```

Ubuntu ships systemd timers for automatic scrubs.

For a home NAS with large spinning disks, I use a monthly scrub:

```bash
sudo systemctl enable --now \
    zfs-scrub-monthly@rpool.timer
```

Verify:

```bash
systemctl list-timers 'zfs-scrub*'
```

A weekly full scrub may be unnecessary for a large mostly-static home NAS, while monthly is a reasonable starting point.

## Disk health monitoring

### SMART monitoring

ZFS and SMART solve different problems.

ZFS sees things like:

```text
READ errors
WRITE errors
checksum errors
```

SMART sees the internal state of the physical drive.

Install `smartmontools`:

```bash
sudo apt install smartmontools
sudo systemctl enable --now smartmontools.service
```

Inspect a disk using a stable by-id path:

```bash
sudo smartctl -x \
    /dev/disk/by-id/ata-DISK
```

For WD HDDs, I pay particular attention to:

```bash
sudo smartctl -A \
    /dev/disk/by-id/ata-DISK \
    | grep -E \
'Power_On_Hours|Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable|UDMA_CRC_Error_Count|Start_Stop_Count|Load_Cycle_Count|Temperature'
```

A healthy new drive should generally start around:

```text
Reallocated_Sector_Ct   0
Current_Pending_Sector  0
Offline_Uncorrectable   0
UDMA_CRC_Error_Count    0
```

A single SMART value is not always enough to condemn a disk.

What matters is the trend together with ZFS errors and SMART self-test results.

### Scheduled SMART self-tests

`smartd` can run regular self-tests.

A simple `/etc/smartd.conf` example:

```text
DEVICESCAN -a -s (S/../.././02|L/../../7/04:003)
```

Short tests start daily in the 02:00–03:00 window, using the NAS's local time.
Long tests are scheduled on Sundays, initially in the 04:00–05:00 window;
`:003` shifts their start windows by three hours for each successive device.
For four detected drives, the long-test windows start at 04:00, 07:00, 10:00
and 13:00. Actual starts depend on the daemon's polling cycle, and long tests
may still overlap if they take more than three hours. This syntax is described
in [smartd.conf(5)](https://manpages.ubuntu.com/manpages/resolute/man5/smartd.conf.5.html).

`DEVICESCAN` applies to all discovered devices, including any supported USB DOM.
Use explicit device entries if only the four pool HDDs should be tested, and
check the resulting schedule below.

After editing the configuration:

```bash
sudo systemctl restart smartmontools
```

Check the schedule:

```bash
sudo smartd -q showtests
```

Check logs:

```bash
journalctl -u smartmontools
```

## Prometheus monitoring and alerting

### Why monitoring matters

RAID without alerts is dangerous.

The failure sequence you want to avoid is:

```text
HDD1 fails
 ↓
RAIDZ2 keeps working
 ↓
nobody notices
 ↓
HDD2 fails weeks later
 ↓
still nobody notices
 ↓
HDD3 fails
 ↓
pool is lost
```

The first degraded state should result in an alert within minutes.

I use Prometheus and Alertmanager for this.

Prometheus itself should preferably run somewhere other than the NAS.

If the NAS loses power completely, an external monitoring server can still detect that it disappeared.

### Installing Prometheus exporters

On the NAS:

```bash
sudo apt install \
    prometheus-node-exporter \
    prometheus-smartctl-exporter
```

Enable them:

```bash
sudo systemctl enable --now \
    prometheus-node-exporter

sudo systemctl enable --now \
    prometheus-smartctl-exporter
```

Check node exporter:

```bash
curl -s localhost:9100/metrics | head
```

Check SMART exporter:

```bash
curl -s localhost:9633/metrics | head
```

### Monitoring ZFS pool state

Node exporter has a ZFS collector.

A particularly useful metric is:

```text
node_zfs_zpool_state
```

Check it:

```bash
curl -s localhost:9100/metrics \
    | grep node_zfs_zpool_state
```

You may see something similar to:

```text
node_zfs_zpool_state{state="online",zpool="rpool"} 1
node_zfs_zpool_state{state="degraded",zpool="rpool"} 0
```

This makes ZFS pool-state alerting straightforward.

### SMART metrics

The SMART exporter exposes metrics such as:

```text
smartctl_device
smartctl_device_smart_status
smartctl_device_status
smartctl_device_smartctl_exit_status
smartctl_device_temperature
smartctl_device_attribute
```

Inspect the actual output generated by the version installed on your system:

```bash
curl -s localhost:9633/metrics \
    | grep smartctl_device_smart_status
```

and:

```bash
curl -s localhost:9633/metrics \
    | grep smartctl_device_temperature
```

Inspect the device inventory and diagnostic status as well:

```bash
curl --fail --silent --show-error localhost:9633/metrics \
    | grep -E '^smartctl_device(_status|_smartctl_exit_status)?\{'
```

I always inspect the real exporter output before writing PromQL rules, because
exact labels and ATA attributes can vary between drive models. Compare health
results with device status and the smartctl exit status when diagnosing a
failure. The latter is a [bitmask](https://manpages.ubuntu.com/manpages/resolute/man8/smartctl.8.html#exit_status),
so a nonzero value does not necessarily mean the command could not read the
device.

In the [upstream metric example](https://github.com/prometheus-community/smartctl_exporter/blob/master/EXAMPLE.md),
`model_name` and `serial_number` are labels of `smartctl_device`, not of every
SMART metric. Verify those labels locally before using the WD model selector
below; replace it if your pool uses different drives.

### Prometheus scrape configuration

Example:

```yaml
scrape_configs:
  - job_name: nas-node
    static_configs:
      - targets:
          - nas:9100

  - job_name: nas-smart
    static_configs:
      - targets:
          - nas:9633
```

### Alert when the NAS disappears

```yaml
- alert: NASDown
  expr: up{job="nas-node"} == 0
  for: 2m

  labels:
    severity: critical

  annotations:
    summary: "NAS is unreachable"
```

This alert must live outside the NAS itself.

### Alert when the SMART exporter disappears

The SMART exporter is a separate scrape target. Node exporter can remain
healthy while SMART monitoring is unavailable:

```yaml
- alert: SMARTExporterDown
  expr: up{job="nas-smart"} == 0
  for: 2m

  labels:
    severity: warning

  annotations:
    summary: "SMART exporter on NAS is unreachable"
```

Keep both targets in the scrape configuration. Removing a target also removes
its `up` series, so these availability alerts no longer cover it.

### Alert when the ZFS pool is not ONLINE

```yaml
- alert: ZFSPoolNotOnline
  expr: |
    (
      node_zfs_zpool_state{
        job="nas-node",
        zpool="rpool",
        state="online"
      } != 1
    )
    or
    (
      up{job="nas-node"} == 1
      unless on (job, instance)
      node_zfs_zpool_state{
        job="nas-node",
        zpool="rpool",
        state="online"
      }
    )

  for: 1m

  labels:
    severity: critical

  annotations:
    summary: "ZFS pool rpool is not ONLINE or is missing"
```

A removed or failed HDD should cause this alert quickly. The second branch
also detects a missing pool-state metric for each reachable `nas-node` target;
`NASDown` covers failed scrapes. The [PromQL `unless` operator](https://prometheus.io/docs/prometheus/latest/querying/operators/#logical-set-binary-operators)
matches by `job` and `instance`, so a healthy NAS cannot hide a missing metric
on another one. These examples assume every target in this job has `rpool`.

### Alert when expected HDD metrics disappear

For this four-WD60EFRX setup, check the number of matching devices **per NAS**.
An ungrouped `count` could combine drives from different machines. If no
matching devices remain, `count` returns no series; the `or` branch below
supplies zero for each configured target:

```yaml
- alert: SMARTPoolDisksMissing
  expr: |
    (
      (
        count by (job, instance) (
          smartctl_device{job="nas-smart", model_name=~".*WD60EFRX.*"}
        )
        or on (job, instance)
        (0 * up{job="nas-smart"})
      ) < 4
    )
    and on (job, instance)
    (up{job="nas-smart"} == 1)

  for: 5m

  labels:
    severity: warning

  annotations:
    summary: "SMART exporter reports fewer than four expected pool HDDs"
```

`SMARTExporterDown` covers an unreachable exporter. This inventory rule checks
only that four matching devices are reported; it does not establish their
health or ZFS membership. If other disks share the same model, select the four
expected serial numbers instead. Update the selector after replacing a disk.

### SMART health alert

```yaml
- alert: SMARTHealthFailed
  expr: |
    smartctl_device_smart_status{job="nas-smart"} == 0

  for: 2m

  labels:
    severity: critical

  annotations:
    summary: "SMART health check failed"
```

This covers every device exposed by the `nas-smart` job. If the exporter also
sees a USB DOM, NVMe drive or other storage, restrict the rule to the pool HDDs
when that is the intended scope. To filter by model or serial, join the health
metric to `smartctl_device` with `and on (job, instance, device)`; adding a
`model_name` filter directly to a metric without that label would hide it.
Missing device series are handled separately by the inventory rule above.

### Alert on bad or pending sectors

Example:

```yaml
- alert: SMARTBadSectors
  expr: |
    smartctl_device_attribute{
      attribute_name=~"Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable",
      attribute_value_type="raw"
    } > 0

  for: 5m

  labels:
    severity: warning

  annotations:
    summary: "SMART reports bad or pending sectors"
```

Depending on your exporter version, inspect the actual metric labels first.

### Alert on growing SATA CRC errors

CRC errors are especially useful as a rate rather than an absolute counter.

Example:

```yaml
- alert: SATACRCErrorIncreasing
  expr: |
    increase(
      smartctl_device_attribute{
        attribute_name="UDMA_CRC_Error_Count",
        attribute_value_type="raw"
      }[1h]
    ) > 0

  labels:
    severity: warning

  annotations:
    summary: "SATA CRC error counter is increasing"
```

A growing CRC counter can point to:

```text
SATA cabling
backplane problems
connector problems
controller issues
```

rather than platter failure.

### HDD temperature alert

Example:

```yaml
- alert: HDDTemperatureHigh
  expr: |
    smartctl_device_temperature{
      temperature_type="current"
    } > 50

  for: 10m

  labels:
    severity: warning

  annotations:
    summary: "HDD temperature is high"
```

The `50°C` value is an example warning threshold.

Use the specification of your actual HDD model when choosing final limits.

## Monitoring the boot ESP replicas

### Manual ESP verification

This NAS has another failure domain that a typical ZFS system does not have:

```text
4 independent ESPs
```

So I also want to know whether they are still synchronized.

Manual verification:

```bash
sudo zbm-esp-sync verify
```

A failed verification returns a non-zero status.

I would not run a full SHA256 verification every minute, especially if HDD spindown is enabled.

Once per day is sufficient.

### Exporting ESP status to Prometheus

Node exporter includes a [textfile collector](https://github.com/prometheus/node_exporter#textfile-collector).
The script below uses `/var/lib/prometheus/node-exporter`, the packaged
collector directory. Verify the service configuration on the installed version:

```bash
systemctl cat prometheus-node-exporter
cat /etc/default/prometheus-node-exporter
```

If you override its arguments, retain
`--collector.textfile.directory=/var/lib/prometheus/node-exporter`. Restart the
exporter after changing them and confirm the metrics appear below.

Create a small monitoring script:

```bash
sudo tee /usr/local/sbin/zbm-esp-monitor >/dev/null <<'EOF'
#!/usr/bin/env bash

set -euo pipefail

DIR=/var/lib/prometheus/node-exporter
OUT="$DIR/zbm-esp.prom"
install -d -m 0755 "$DIR"
TMP=$(mktemp "$DIR/.zbm-esp.prom.XXXXXX")
trap 'rm -f "$TMP"' EXIT

if /usr/local/sbin/zbm-esp-sync verify >/dev/null 2>&1; then
    ok=1
else
    ok=0
fi

{
    echo '# HELP zbm_esp_sync_ok ZFSBootMenu ESP replicas match the master'
    echo '# TYPE zbm_esp_sync_ok gauge'
    echo "zbm_esp_sync_ok $ok"

    echo '# HELP zbm_esp_sync_last_check_timestamp_seconds Last ESP verification time'
    echo '# TYPE zbm_esp_sync_last_check_timestamp_seconds gauge'
    echo "zbm_esp_sync_last_check_timestamp_seconds $(date +%s)"
} > "$TMP"

chmod 0644 "$TMP"
mv "$TMP" "$OUT"
EOF

sudo chmod 0755 \
    /usr/local/sbin/zbm-esp-monitor
```

Verify the actual binary location:

```bash
command -v zbm-esp-sync
```

and adjust the script if required.

### Daily ESP verification with systemd

Create a service:

```bash
sudo tee \
  /etc/systemd/system/zbm-esp-monitor.service \
  >/dev/null <<'EOF'
[Unit]
Description=Verify ZFSBootMenu EFI replicas

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/zbm-esp-monitor
EOF
```

Create a timer:

```bash
sudo tee \
  /etc/systemd/system/zbm-esp-monitor.timer \
  >/dev/null <<'EOF'
[Unit]
Description=Periodic ZFSBootMenu ESP verification

[Timer]
OnCalendar=*-*-* 06:15:00
Persistent=true

[Install]
WantedBy=timers.target
EOF
```

Enable it:

```bash
sudo systemctl daemon-reload

sudo systemctl start zbm-esp-monitor.service
sudo systemctl enable --now \
    zbm-esp-monitor.timer
```

Check:

```bash
systemctl list-timers \
    zbm-esp-monitor.timer
```

Verify the metrics:

```bash
curl -s localhost:9100/metrics \
    | grep '^zbm_esp'
```

Expected output:

```text
zbm_esp_sync_ok 1
zbm_esp_sync_last_check_timestamp_seconds ...
```

### Alert when ESPs are out of sync

```yaml
- alert: ZFSBootMenuESPsOutOfSync
  expr: zbm_esp_sync_ok != 1

  for: 5m

  labels:
    severity: warning

  annotations:
    summary: "ZFSBootMenu ESP replicas are not synchronized"
```

Also monitor whether the check itself has stopped running:

```yaml
- alert: ZFSBootMenuESPCheckStale
  expr: |
    (time() - zbm_esp_sync_last_check_timestamp_seconds{job="nas-node"} > 90000)
    or
    (up{job="nas-node"} == 1
      unless on (job, instance)
      zbm_esp_sync_last_check_timestamp_seconds{job="nas-node"})

  labels:
    severity: warning

  annotations:
    summary: "ZFSBootMenu ESP verification is stale or missing"
```

The second branch detects a missing metric for each healthy NAS scrape target.
`NASDown` handles targets that cannot be scraped. Keep the NAS target in the
Prometheus configuration: removing a target also removes its `up` series.

## ZFS Event Daemon

I also keep ZED enabled:

```bash
sudo systemctl enable --now \
    zfs-zed.service
```

Inspect events manually:

```bash
sudo zpool events -v
```

ZED gives another independent channel for reacting to ZFS events.

The monitoring stack therefore looks like this:

```text
                 NAS
                  │
        ┌─────────┼─────────┐
        │         │         │
       ZFS       SMART     ESPs
        │         │         │
 node_exporter    │    zbm-esp-monitor
        │         │         │
        └─────────┼─────────┘
                  │
              Prometheus
                  │
             Alertmanager
                  │
           email / Telegram
```

And independently:

```text
ZFS kernel
    ↓
   ZED
```

## Power management

### Suspend and Wake-on-LAN

Ubuntu Server can suspend if the platform supports it.

Check available power states:

```bash
cat /sys/power/state
cat /sys/power/mem_sleep
```

Suspend manually:

```bash
sudo systemctl suspend
```

For network wake-up, use Wake-on-LAN.

Install `ethtool`:

```bash
sudo apt install ethtool
```

Check NIC capabilities:

```bash
sudo ethtool enp1s0 \
    | grep -i Wake-on
```

For example:

```text
Supports Wake-on: pumbg
Wake-on: d
```

If `g` is present, Magic Packet wake-up is supported.

Enable it:

```bash
sudo ethtool -s enp1s0 wol g
```

Netplan can keep the setting persistent:

```yaml
network:
  version: 2
  renderer: networkd

  ethernets:
    enp1s0:
      dhcp4: true
      wakeonlan: true
```

There is an important limitation here.

A sleeping server generally will not wake merely because somebody tries to open an SMB share.

The usual workflow is:

```text
client sends WoL Magic Packet
        ↓
NAS wakes up
        ↓
Samba/SSH becomes available
```

For a NAS that needs to remain reachable at all times, normal idle may be preferable to full suspend.

I would also avoid overly aggressive HDD spindown, because SMART, monitoring, ZFS metadata updates and snapshots may repeatedly wake the drives and generate excessive start/stop cycles.

## Final architecture

The final boot architecture is:

```text
                    TerraMaster
                         │
                ┌────────┴────────┐
                │                 │
              Boot               Data
                │                 │
         4 independent ESPs      rpool
                │                 │
         ZFSBootMenu            RAIDZ2
                │          ┌──────┼──────┐
         zbm-esp-sync       │      │      │
                │         HDD1   HDD2   HDD3/HDD4
                │
       automatic force import
                │
        rpool/ROOT/ubuntu
```

Operationally:

```text
RAIDZ2
  │
  ├── monthly scrub
  │
  ├── SMART monitoring
  │     ├── short tests
  │     └── staggered long tests
  │
  ├── datasets
  │     ├── shared
  │     ├── documents
  │     ├── media
  │     └── backups
  │
  ├── Sanoid snapshots
  │
  ├── node_exporter
  │
  ├── smartctl_exporter
  │
  ├── ESP monitoring
  │
  ├── ZED
  │
  └── Prometheus + Alertmanager
```

## What RAIDZ2 does not solve

RAIDZ2 is not a backup.

It protects against disk failures.

It does not protect against:

```text
accidental deletion
ransomware
application corruption
administrator mistakes
NAS theft
fire
power supply failure affecting several drives
complete pool loss
```

Snapshots help with some logical failures, but they still live on the same pool.

A complete design should eventually look like:

```text
              Primary NAS
                  │
        RAIDZ2 + snapshots
                  │
           ZFS replication
                  │
                  ▼
        separate physical NAS
          or backup server
```

For a ZFS-based system, `zfs send` / `zfs receive` or Syncoid are natural choices.

## Conclusion

My original goal was to build a NAS that does not rely on a separate system SSD and can survive the loss of any two HDDs.

The result is a system where Ubuntu itself lives inside the RAIDZ2 pool, while every disk contains an independently bootable EFI System Partition.

ZFS provides data redundancy.

ZFSBootMenu provides a clean root-on-ZFS boot path.

Four ESPs eliminate the single boot-device failure point.

`zbm-esp-sync` keeps the boot partitions synchronized.

`zbm.import_policy=force` allows this standalone NAS to import a degraded pool without manual recovery.

Monthly scrubs detect latent corruption.

SMART monitors physical drive health.

Sanoid provides snapshot retention.

Prometheus and Alertmanager make sure that the first failed disk does not remain unnoticed.

And there is no need to build a RAM disk for ZFS caching, because ARC already provides the correct in-memory cache layer.

Most importantly, this was not only tested in theory.

The NAS successfully booted in all of these states:

```text
4 disks present
    → ONLINE

1 disk missing
    → DEGRADED
    → boot successful

2 disks missing
    → DEGRADED
    → boot successful
```

With two physical HDDs removed, the machine still booted from an ESP on one of the remaining disks, ZFSBootMenu imported the RAIDZ2 pool automatically, Ubuntu mounted `rpool/ROOT/ubuntu`, networking came up, and SSH became available without manual intervention.

That is exactly the failure mode I wanted: losing two disks should result in a degraded NAS that needs maintenance, not an unbootable server that needs a rescue USB.