Liborator Articles About

Published: 4th October 2026

Installing FreeBSD 15 on Raspberry Pi 4B with ZFS root

I am running several Raspberry Pi 4B machines. They are wonderful for hosting DNS, samba, network diagnostics and other services.

Recently though, one of them started acting really strange. I tracked it down to the microSD card being fried and switching itself to read-only mode (to prevent data loss), yet staying completely silent about it.

I do not want this happening again. So I made my RPi boot directly from SSD with the amazing ZFS file system, configuration known as Root-on-ZFS.

This article documents how to do it. I

  1. introduce the script driving the installation
  2. describe how to prepare your computer supporting the installation
  3. describe how to prepare the Raspberry Pi for installation
  4. comment on various parts of the installation
  5. describe ancillary utilities for documentation
  6. reflect on the journey and what could be done differently
  7. provide the complete script

Let’s get to it!

Disclaimer: It’s a shame I need to even mention it, but the year is 2026 and folks are trying to cut corners with their (ab)use of generative AI. This article is written by me and me only.

I prompted myself to “Make no mistakes.”. But I am just a guy on the Internet, so double-check everything.

Attribution: I sincerely thank Nick Avgerinos for their article, which is about the same topic.

My contributions and additions are creating a standalone, configurable, documented script of the installation procedure, together with some fixes. The script also includes some neat techniques and tricks I regularly use.1

§ Script introduction

Download the rpi-rootzfs-install.sh script.
(It’s copy-pasted at the bottom of this article, so you don’t have to open your text editor to view it)

The installation procedure assumes FreeBSD system, but you can run the help command anywhere.

$ sh rpi-rootzfs-install.sh help
rpi-rootzfs-install.sh - Install Root-on-ZFS on a disk for Raspberry Pi 4B

Following functions can be executed:

help
    Print help, public function documentation and environment variables used
    to configure the script.

print_function_docs
    Print all public functions and their documentation.

print_config_vars_docs
    Print all 'RPIZFS_' variables and their documentation that may be configured
    by the user.

print_config_values
    Print the values of all 'RPIZFS_' variables that be configured by the user.
    They are in shell format - "name=value" without spaces - so can be dumped
    to a file and then sourced from it again.

check_system
    Check necessary preconditions of the installation, like
    existence of the disk, permissions, etc.

install
    Check system and then execute the whole installation procedure.

---

Following environment variables can be used to configure the script behaviour:

RPIZFS_DISK
    Name of the disk that will have the FreeBSD on ZFS installed.

RPIZFS_IMG
    FreeBSD image to install.

RPIZFS_POOLMOUNTPOINT
    Where to mount the ZFS pool on the newly installed FreeBSD.

RPIZFS_POOLNAME
    Name of the ZFS pool on the newly installed FreeBSD.

RPIZFS_REMOTE_URL
    Where to download the decompressed FreeBSD image from. May be left as dummy value when
    the 'RPIZFS_IMG' image is already present in the 'RPIZFS_TMPDIR'.

RPIZFS_TMPDIR
    Where to download to or look for the FreeBSD image. Please note that '/tmp' will work only on
    Raspberries that have more than ~5GB RAM, which is the size of the downloaded image! That is
    because by default, '/tmp' directory is mounted as a temporary filesystem (tmpfs), so
    contents are held entirely in RAM.

After preparation, what is to be executed is function install:

install() {
    # Check system and then execute the whole installation procedure.
    echo "Installation procedure initialized."
    echo "Variables used:"
    print_config_values

    echo "Checking system:"
    check_system

    # Fetch the uncompressed FreeBSD installation image ...
    _fetch_image
    # ... burn it to the external disk ...
    _dd_to_disk
    # ... tweak the disk partitions ...
    _resize_partitions
    # ... create a filesystem hierarchy in a new ZFS pool on the external disk ...
    _create_rootzfs
    # ... copy the FreeBSD installation image to the filesystem in the pool ...
    _copy_image_to_rootzfs
    # ... adjust '/boot/loader.conf' and `/etc/fstab`, so RPi boots correctly from ZFS ...
    _adjust_boot_conf
    # ... create '/etc/rc.conf' on RPi ...
    _create_rc_conf
    # ... and finally, unmount the ZFS pool.
    _export_zpool

    echo
    echo "Installation successful."
    echo "You can shutdown the RPi ('shutdown now'), remove the microSD card and start it again."
}

invoked as:

root@rpi# sh rpi-rootzfs-install.sh install

or, if customization is desired:

root@rpi# RPIZFS_REMOTE_URL=http://192.168.123.123:8080 sh \
    rpi-rootzfs-install.sh \
    install

In the rest of the article, I will comment on the functions above and machinery behind automatic function and environment variable documentation.

Pro Tip: If you want to trace what the script does, you can launch sh with -x option, which prints each command actually executed prefixed by + to standard error:

$ sh -x rpi-rootzfs-install.sh help
# Redirect stderr containing traces for later inspection to a file.
$ sh -x rpi-rootzfs-install.sh help 2>/tmp/program-trace.log

§ Preparation of your PC

You will another computer besides the Raspberry Pi (named other-pc where distinction is relevant).

Download a suitable image from FreeBSD Handbook - Installing FreeBSD. As of writing, that’s FreeBSD-15.1-RELEASE-arm64-aarch64-RPI.img.xz.

Decompress the image and burn it to a microSD card. Assuming the microSD has name dx0, you burn it using dd:

root@other-pc# dd \
    if=FreeBSD-15.1-RELEASE-arm64-aarch64-RPI.img \
    of=/dev/dx0 \
    bs=4M \
    status=progress

You will also need a way to run a HTTP server serving files. I use Python for this, because the http.server module is part of standard library.

We already downloaded the image from FreeBSD public mirror. Therefore, not to waste their bandwith again, we will download it from the other computer running the HTTP server instead (_fetch_image function in the script).

The shell script containing the installation procedure will also be downloaded from the HTTP server.

Start the HTTP server in a directory that contains the image and the shell script:

you@other-pc$ python3 -m http.server -b 0.0.0.0 8080

This command starts the HTTP server listing contents of the directory on all interfaces and binds it to port 8080. Note the LAN address of the computer, it will be needed later:

# BSDs / macOS
you@other-pc $ ifconfig
# Debian / Ubuntu / others making the mistake of abandoning ifconfig in favour
# of ip so we have to relearn everything...
you@other-pc $ ip a

§ Preparation of the Raspberry Pi 4B

Boot up the RPi with microSD card inside. You can either connect to it via ssh on your LAN or connect micro HDMI cable and keyboard and log in. By default, credentials are freebsd:freebsd and root:root.

Once you are logged in as root on the RPi, connect the external disk and run:

root@rpi:/tmp# geom disk list

You should see the microSD card, its partitions and the external disk.

Download the installation script from the other computer running the HTTP server, with IP address obtained in previous section.

freebsd@rpi:/tmp fetch http://<address_and_port_of_the_other_pc>/rpi-rootzfs-install.sh

To see the current configuration, run the print_config_values command:

freebsd@rpi:/tmp $ sh rpi-rootzfs-install.sh print_config_values
RPIZFS_DISK=da0
RPIZFS_IMG=FreeBSD-15.1-RELEASE-arm64-aarch64-RPI.img
RPIZFS_POOLMOUNTPOINT=/zroot
RPIZFS_POOLNAME=zroot
RPIZFS_REMOTE_URL=http://192.0.2.1:8080
RPIZFS_TMPDIR=/tmp

You can verify RPi is ready for installation by check_system command (don’t worry, it is run in the install command as well). Following shows 2 early exits - invocation by a regular user and referencing a nonexistent disc - and a success:

freebsd@rpi:/tmp$ sh rpi-rootzfs-install.sh check_system
Checking existence of '/tmp/FreeBSD-15.1-RELEASE-arm64-aarch64-RPI.img' ... will need to be downloaded.
Checking existence of 'da0' disk ... OK
Checking ability to write to '/mnt' directory ... not writable! Exiting.
freebsd@rpi:/tmp$ echo $?
1

freebsd@rpi:/tmp$ RPIZFS_DISK=oh-no sh rpi-rootzfs-install.sh check_system
Checking existence of '/tmp/FreeBSD-15.1-RELEASE-arm64-aarch64-RPI.img' ... will need to be downloaded.
Checking existence of 'oh-no' disk ... does not exist! Exiting.
freebsd@rpi:/tmp$ echo $?
1

root@rpi:/tmp$ sh rpi-rootzfs-install.sh check_system
Checking existence of '/tmp/FreeBSD-15.1-RELEASE-arm64-aarch64-RPI.img' ... will need to be downloaded.
Checking existence of 'da0' disk ... OK
Checking ability to write to '/mnt' directory ... OK
root@rpi:/tmp$ echo $?
0

If everything goes well, you are now ready to run (as root) the installation command:

root@rpi:/tmp# sh rpi-rootzfs-install.sh install

If you are successful, you can turn the RPi off, remove the microSD card and start RPi again.

And that’s it! You should have a working Raspberry Pi booting from an SSD running ZFS filesystem now.

In the rest of the article, I will comment on parts of the installation process and ancillary utilities. If you aren’t interested, you can jump straight to Closing remarks.

§ Comments on the installation procedure

Note: To declutter the code, I expand the global variables to their default values and place them at the beginning of the snippets.

§ Beginning of the installation
RPIZFS_DISK="da0"
RPIZFS_IMG="FreeBSD-15.1-RELEASE-arm64-aarch64-RPI.img"
RPIZFS_REMOTE_URL="http://192.0.2.1:8080"

rpizfs_diskpath="/dev/da0"
rpizfs_imgpath="/tmp/FreeBSD-15.1-RELEASE-arm64-aarch64-RPI.img"
rpizfs_partition="da0s2"

_fetch_image() {
    # Fetch the installation image. No-op if the image is already present.
    if test -f "${rpizfs_imgpath}"; then
        echo "Image already downloaded. Skipping"
    else
        local remote_image_uri="${RPIZFS_REMOTE_URL}/${RPIZFS_IMG}"
        fetch "${remote_image_uri}" -o "${rpizfs_imgpath}"
    fi
}

_dd_to_disk() {
    # Burn the FreeBSD install image to the disk.
    dd \
        if="${rpizfs_imgpath}" \
        of="${rpizfs_diskpath}" \
        bs=4M \
        iflag=direct \
        oflag=direct \
        status=progress
}

_resize_partitions() {
    # Resize and modified partitions created by the '_dd_to_disk' function.
    gpart resize -i 2 "${RPIZFS_DISK}"
    gpart modify -i 1 "${rpizfs_partition}"
    gpart resize \
        -i 1 \
        -a 1M \
        "${rpizfs_partition}"
}

Functions _fetch_image, _dd_to_disk and _resize_partitions are pretty self-explanatory.

Even though we will overwrite the actual filesystem later with the ZFS layer, we still want to create the partition table and have the boot partition (da0s1 if disk’s name is da0) present.

I also want the actual filesystem partition - da0s2 - to be aligned to 1MB (gpart resize -a 1M ...). I believe any even multiple of the ashift property would be fine (in our case, 2 ^ 12 = 4096 = 4KB, see below), but let’s keep 1MB.

§ Functions _create_rootzfs and _copy_image_to_rootzfs
RPIZFS_POOLMOUNTPOINT="/zroot"
RPIZFS_POOLNAME="zroot"

rpizfs_altroot="/rpizfs-altroot"
rpizfs_imgpath="/tmp/FreeBSD-15.1-RELEASE-arm64-aarch64-RPI.img"
rpizfs_mntdir="/mnt"
rpizfs_partition="da0s2"

_create_rootzfs() {
    # Create a ZFS pool and the filesystem hierarchy within it.
    :
    # https://openzfs.github.io/openzfs-docs/man/master/8/zpool-create.8.html
    zpool create \
        -f \
        -O compress=lz4 \
        -O atime=off \
        -o ashift=12 \
        -o altroot="${rpizfs_altroot}" \
        -m none \
        "${RPIZFS_POOLNAME}" \
        "${rpizfs_partition}"

    # https://openzfs.github.io/openzfs-docs/man/master/8/zfs-create.8.html
    zfs create -o mountpoint=none "${RPIZFS_POOLNAME}/ROOT"
    zfs create -o mountpoint=/ "${RPIZFS_POOLNAME}/ROOT/default"
    zfs create -o mountpoint=/tmp -o exec=on -o setuid=off "${RPIZFS_POOLNAME}/tmp"
    zfs create -o mountpoint=/usr -o canmount=off "${RPIZFS_POOLNAME}/usr"
    zfs create -o setuid=off "${RPIZFS_POOLNAME}/usr/ports"
    zfs create "${RPIZFS_POOLNAME}/usr/src"
    zfs create -o mountpoint=/var -o canmount=off "${RPIZFS_POOLNAME}/var"
    zfs create -o exec=off -o setuid=off "${RPIZFS_POOLNAME}/var/audit"
    zfs create -o exec=off -o setuid=off "${RPIZFS_POOLNAME}/var/crash"
    zfs create -o exec=off -o setuid=off "${RPIZFS_POOLNAME}/var/log"
    zfs create -o atime=on "${RPIZFS_POOLNAME}/var/mail"
    zfs create -o setuid=off "${RPIZFS_POOLNAME}/var/tmp"

    zfs set mountpoint="${RPIZFS_POOLMOUNTPOINT}" "${RPIZFS_POOLNAME}"
    zpool set bootfs="${RPIZFS_POOLNAME}/ROOT/default" "${RPIZFS_POOLNAME}"
    zfs set canmount=noauto "${RPIZFS_POOLNAME}/ROOT/default"
}

_copy_image_to_rootzfs() {
    # Create a virtual disk from the install image and copy it to the filesystem
    # created by the '_create_rootzfs' function.
    local md_disk_name="md123";                        readonly md_disk_name
    local md_partition_disk="/dev/${md_disk_name}s2a"; readonly md_partition_disk
    local mnt_dir="${rpizfs_mntdir}/rpi-rootzfs";      readonly mnt_dir

    # Create a "virtual disk" from the install image.
    # https://man.freebsd.org/cgi/man.cgi?mdconfig
    mdconfig -a -t vnode -f "${rpizfs_imgpath}" -u "${md_disk_name}"

    mkdir -p "${mnt_dir}"
    mount "${md_partition_disk}" "${mnt_dir}"

    # The critical step - the mounted install image is archived in its entirety,
    # and then unarchived in the ZFS file system hierarchy created by the
    # '_create_rootzfs' function.
    ( cd "${mnt_dir}" && tar cf - . ) | ( cd "${rpizfs_altroot}" && tar xf - )

    # Clean-up: Unmount the virtual disk, destroy it and remove the mount
    #   directory.
    umount "${mnt_dir}"
    mdconfig -d -u "${md_disk_name}"
    rm -rf "${mnt_dir}"
}

These 2 functions are the most important.

_create_rootzfs function creates a ZFS pool with zroot name on the da0s2 partition. Pool is mounted on /rpizfs-altroot directory, so it does not clash with the current filesystem.

Afterwards, ZFS datasets mimicking the FreeBSD filesystem are created in the pool. Some datasets have hardening properties like setuid=off.

Last 3 lines of the _create_rootzfs ensure that on next boot of the RPi, it knows where to find the bootable dataset via the bootfs zpool property.

In _copy_image_to_rootzfs function the mdconfig is used to create a virtual disc from the installation image. The disc is then mounted and copied in its entirety to the ZFS filesystem created in the _create_rootzfs function.

Archiving to stdout (tar cf -) and subsequent unarchiving from stdout (tax xf -) is a pretty neat pattern. Both are done in a subshell because of the enclosing () brackets, so the directory will not change after execution of the whole script.

However, I was unable to decipher why we need to dd the whole installation image in the _dd_to_disk function, when it would be sufficient to copy just the first ~50MB containing the partition table and the boot partition da0s1?

If you have any idea why it’s done, please, send me an email, it’s in the About section.

§ Post-copying adjustments
rpizfs_altroot="/rpizfs-altroot"

_adjust_boot_conf() {
    # Adjust configuration so the newly installed Raspberry Pi boots correctly
    # on ZFS.
    :
    # https://man.freebsd.org/cgi/man.cgi?loader.conf
    echo 'zfs_load="YES"' >> "${rpizfs_altroot}/boot/loader.conf"
    # Following allows time synchronization running as a non-root user.
    # https://man.freebsd.org/cgi/man.cgi?query=mac_ntpd&sektion=4&format=html
    echo 'mac_ntpd_load="YES"' >> "${rpizfs_altroot}/boot/loader.conf"
    # I enable Wireguard kernel module by default.
    # https://man.freebsd.org/cgi/man.cgi?query=if_wg&sektion=4&format=html
    echo 'if_wg_load="YES"' >> "${rpizfs_altroot}/boot/loader.conf"
    # Comment out mounting UFS file system.
    sed -i '' '/ufs/ s/^/#/' "${rpizfs_altroot}/etc/fstab"
    # Comment out mounting tmpfs on '/tmp' directory. There is a separate ZFS
    # dataset for it.
    sed -i '' '/tmpfs/ s/^/#/' "${rpizfs_altroot}/etc/fstab"
}

_create_rc_conf() {
    # Create a basic '/etc/rc.conf' for the newly installed Raspberry Pi.
    :
    # https://man.freebsd.org/cgi/man.cgi?rc.conf
    echo 'hostname="rpi-rootzfs"' > "${rpizfs_altroot}/etc/rc.conf"
    echo 'sshd_enable="YES"' >> "${rpizfs_altroot}/etc/rc.conf"
    echo 'sendmail_enable="NONE"' >> "${rpizfs_altroot}/etc/rc.conf"
    echo 'ifconfig_DEFAULT="DHCP"' >> "${rpizfs_altroot}/etc/rc.conf"
    # https://docs.freebsd.org/en/books/handbook/config/#configtuning-syslog
    # That's the default, but have it here anyway.
    echo 'syslogd_flags="-s"' >> "${rpizfs_altroot}/etc/rc.conf"

    # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
    echo 'dumpdev="NO"' >> "${rpizfs_altroot}/etc/rc.conf"
    echo 'powerd_enable="YES"' >> "${rpizfs_altroot}/etc/rc.conf"
    echo 'zfs_enable="YES"' >> "${rpizfs_altroot}/etc/rc.conf"

    # If the 'mac_ntpd_load' is loaded, this enables running the 'ntpd' daemon
    # as a non-root user and synchronizes time on startup.
    echo 'ntpd_enable="YES"' >> "${rpizfs_altroot}/etc/rc.conf"
    echo 'ntpd_sync_on_start="YES"' >> "${rpizfs_altroot}/etc/rc.conf"
    echo 'ntpd_user="ntpd"' >> "${rpizfs_altroot}/etc/rc.conf"
}

In _adjust_boot_conf, zfs, mac_ntpd and if_wg kernel modules are loaded.

zfs is of course needed. I load the mac_ntpd so ntpd daemon can run as a non-root user and if_wg so Wireguard network interfaces may be created and managed (topic for a future article).

Adjustments to /etc/fstab - file containing filesystems and their mountpoints - are needed. This is the default:

freebsd@rpi $ cat /etc/fstab
# Custom /etc/fstab for FreeBSD embedded images
/dev/ufs/rootfs     /       ufs rw      1   1
/dev/msdosfs/EFI        /boot/efi   msdosfs rw,noatime  0   0
tmpfs           /tmp        tmpfs   rw,mode=1777    0   0
/dev/label/growfs_swap  none        swap    sw      0   0

so we comment out the UFS line as well as the tmpfs, because both will be mounted by the ZFS pool.

_create_rc_conf configures enabled/disabled services. Last 3 lines are there to start the ntpd daemon used for time synchronization as a non-root user and also synchronize the clock at system startup.
That is important for tasks after reboot, when you definitely want to install the latest patches:

root@rpi# freebsd-update fetch install

Having incorrect system time could result in a failure.

And we are done with the actual installation commands, well done! What follows is a description of the script utilities, so again, you are free to jump to Closing remarks if you aren’t interested.

§ Description of ancillary utilities

In most of my shell scripts, I use the “$0 dispatch pattern”. Last line of the script looks like:

"$@"

That means the first argument of the script is always treated as a function to be executed with the rest of the arguments.

$ sh rpi-rootzfs-install.sh xD
rpi-rootzfs-install.sh: ...: xD: not found
§ Function documentation generation

Besides a preamble, I want help to print a documentation of “public” (without leading underscore) functions of the script.

To do it, I structure function documentation similarly to Python docstrings - comments on lines immediately following the function name.

Function print_function_docs invokes the following awk program:

BEGIN {
    in_function_body = 0;
}

# Public function header encountered, so print its name and start accepting comments.
/^[a-zA-Z][a-zA-Z0-9_]+\(\) \{/ {
    in_function_body = 1;
    sub(/\(\) \{/, "", $0);
    print $0;
}

# Non-comment line in function body encountered, so reset state.
/^[ ]+[a-zA-Z0-9_:]+/ {
    if (in_function_body == 1) {
        in_function_body = 0;
        printf "\n";
    }
}

# Comment line in function body encountered, so strip the leading '#' and print it.
/^[ ]*\# / {
    if (in_function_body == 1) {
        sub(/^[ ]*\# /, "    ", $0);
        print $0;
    }
}

# End of function body encountered, so reset state.
/^\}$/ {
    if (in_function_body == 1) {
        in_function_body = 0;
        printf "\n";
    }
}
EOF

For the uninitiated, awk program is, in its basic form, a sequence of pattern {action} pairs. It reads an input file and groups it into logical “Records” that have “Fields”. By default, records are separated by a newline and fields by an empty space. Each record is then matched against all patterns defined by the program and on a successful match, corresponding actions are executed.

In awk, // denote regular expressions and numbered variables $0, $1, $2, … are populated by the whole record in the case of $0 and first, second, … field of the record in the case of higher ordinals.

To become enlightened, you are encouraged to study the manpage of awk.

Program above is thus an automaton which, once it recognizes a “public” function, accepts lines starting with # and prints them indented. Encountering anything else makes it accept public function names again.

§ Configuration variables documentation generation

It’s great when command-line programs announce all the ways you can configure them. I chose configuration via environment variables, since parts of the script reference the same variables, which is clumsy to do with positional arguments only.

I definitely want to “namespace” environment variables used by one script with a common prefix, so RPIZFS_ is chosen.

At the beginning of the script, a section marked by:

# --- BEGIN ENV VARS ---
#
# ...
#
# --- END ENV VARS ---

contains all environment variables with a default parameter expansion:

VAR="${VAR:-123}"

where :- expansion reads roughly as:

If the variable VAR is unset or null, assign to it the default (here “123”). If VAR is set, no-op.

To read all RPIZFS_ variables and their documentations, which is placed right above their declaration, function print_config_vars_docs invokes the following awk program:

BEGIN {
    comment_acc = "";
    in_env_section = 0;
}

# Comment encountered. If we are in the env section, append the line to
# the 'comment_acc' without the leading '#'.
/^# [a-zA-Z0-9\-_]/ {
    if (in_env_section == 1) {
        sub(/^# /, "    ", $0);
        # Yeah, following is string concatenation in awk...
        comment_acc = comment_acc $0 "\n";
    }
}

# Special beginning marker encountered, accumulation can start.
$0 == "# --- BEGIN ENV VARS ---" {
    in_env_section = 1;
}

# Special ending marker encountered, no point in continuing further.
$0 == "# --- END ENV VARS ---" {
    exit 0;
}

# Variable assignment encountered. Print its name and all preceding comment
# lines accumulated. Reset the accumulator.
/^RPIZFS_[a-zA-Z0-9-_]+\=/ {
    sub(/\=.*$/, "", $0);
    print $0;
    print comment_acc;
    comment_acc = "";
}

Program above is again an automaton which, once in the marked section, accepts lines starting with # and accumulates them. Encountering RPIZFS_ variable declaration causes it to print the full name of the variable and all the accumulated comment lines. Doing so resets the accumulator. Actions are repeated until the end of the marked section is reached.

§ Configuration variables values printing

Last utility function is print_config_values:

print_config_values() {
    # Print the values of all 'RPIZFS_' variables that be configured by the user.
    # They are in shell format - "name=value" without spaces - so can be dumped
    # to a file and then sourced from it again.
    for varname in $(grep -E -o -e '^RPIZFS_\w+' "${THIS}"); do
        echo -n "$varname="
        eval echo "\$${varname}"
    done
}

I am slightly disturbed by the use of eval, but I need to first expand the variable name held in "${varname}" - "RPIZFS_DISK/POOLNAME/..., and then expand again for the value of the actual variable.

If shell had dictionaries and made environment variable accessible via ENVIRON, it would be equivalent to:

# Hypothetical
echo "$ENVIRON[$varname]"

§ Closing remarks

If you made it this far, thank you for your attention! I sincerely hope this article helped you and you learnt something new reading it.

It’s a bummer that installing Root-On-ZFS on a Raspberry Pi requires this kind of workaround and is not supported out of the box. Maybe bsdinstall will be runnable on RPis in the future?

Arguably, sizable portion of the script is unnecessary for the task at hand, which is having RPi boot from ZFS on an external SSD.

However, I routinely come back to my article about samba (update coming soon) and copy snippets from it. My intention is the same here - learnings condensed into an article meant for sharing and recall.

In normal settings, awk programs would each get their own file located next to the main script:

# Absolute path to this script.
THIS="$(realpath $0)"
+ # Absolute path to the directory containing this script.
+ HERE="$(dirname ${THIS})"

print_function_docs() {
    # Print all public functions and their documentation.
-   awk "${function_docs_awk}" "${THIS}"
+   awk -f "${HERE}/function_docs.awk" "${THIS}"
}

They are inlined because in terms of simplicity, nothing beats copying a single file.

And … maybe I am straying too far from the topic, but…

If I was to design an extendable system where users can register their own commands and environment variables for configuration, I would need to do things a bit differently.

Suppose I have RPIZFS_PLUGINS_DIR variable, a directory where all plugins - shell scripts - are located.

Then a minimal change would look like:

+ RPIZFS_PLUGINS_DIR="${RPIZFS_PLUGINS_DIR:-/etc/your-plugins}"
+ RPIZFS_LOAD_PLUGINS="${RPIZFS_LOAD_PLUGINS:-1}"

+ load_plugins() {
+    for fname in "${RPIZFS_PLUGINS_DIR}"/*.sh; do
+        source "${fname}"
+    done;
+ }

print_config_vars_docs() {
    # Print all 'RPIZFS_' variables and their documentation that may be configured
    # by the user.
-   awk "${config_vars_awk}" "${THIS}"
+   awk "${config_vars_awk}" "${THIS}" "${RPIZFS_PLUGINS_DIR}"/*.sh
}

and that’s it! Shell paramemter expansion with the *.sh takes care of plugin discovery.

§ Complete script

# Installation script for Root-on-ZFS on a disk for Raspberry Pi 4B
#
# For educational purposes only.
# Author is not liable for any damages caused by incorrect handling of this tool.
#
# Author: Libor Martinek

set -eu

# Absolute path to this script.
THIS="$(realpath $0)"

# --- BEGIN ENV VARS ---
#
# Name of the disk that will have the FreeBSD on ZFS installed.
RPIZFS_DISK="${RPIZFS_DISK:-da0}"
# FreeBSD image to install.
RPIZFS_IMG="${RPIZFS_IMG:-FreeBSD-15.1-RELEASE-arm64-aarch64-RPI.img}"
# Where to mount the ZFS pool on the newly installed FreeBSD.
RPIZFS_POOLMOUNTPOINT="${RPIZFS_POOLMOUNTPOINT:-/zroot}"
# Name of the ZFS pool on the newly installed FreeBSD.
RPIZFS_POOLNAME="${RPIZFS_POOLNAME:-zroot}"
# Where to download the decompressed FreeBSD image from. May be left as dummy value when
# the 'RPIZFS_IMG' image is already present in the 'RPIZFS_TMPDIR'.
RPIZFS_REMOTE_URL="${RPIZFS_REMOTE_URL:-"http://192.0.2.1:8080"}"
# Where to download to or look for the FreeBSD image. Please note that '/tmp' will work only on
# Raspberries that have more than ~5GB RAM, which is the size of the downloaded image! That is
# because by default, '/tmp' directory is mounted as a temporary filesystem (tmpfs), so
# contents are held entirely in RAM.
RPIZFS_TMPDIR="${RPIZFS_TMPDIR:-/tmp}"
#
# --- END ENV VARS ---

# Private or derived variables:
rpizfs_altroot="/rpizfs-altroot";                readonly rpizfs_altroot
rpizfs_diskpath="/dev/${RPIZFS_DISK}";           readonly rpizfs_diskpath
rpizfs_imgpath="${RPIZFS_TMPDIR}/${RPIZFS_IMG}"; readonly rpizfs_imgpath
rpizfs_mntdir="/mnt";                            readonly rpizfs_mntdir
rpizfs_partition="${RPIZFS_DISK}s2";             readonly rpizfs_partition

help() {
    # Print help, public function documentation and environment variables used
    # to configure the script.
    echo "$(basename ${THIS}) - Install Root-on-ZFS on a disk for Raspberry Pi 4B"
    echo
    echo "Following functions can be executed:"
    echo
    print_function_docs
    echo "---"
    echo
    echo "Following environment variables can be used to configure the script behaviour:"
    echo
    print_config_vars_docs
}

function_docs_awk=$(cat <<'EOF'
BEGIN {
    in_function_body = 0;
}

# Public function header encountered, so print its name and start accepting comments.
/^[a-zA-Z][a-zA-Z0-9_]+\(\) \{/ {
    in_function_body = 1;
    sub(/\(\) \{/, "", $0);
    print $0;
}

# Non-comment line in function body encountered, so reset state.
/^[ ]+[a-zA-Z0-9_:]+/ {
    if (in_function_body == 1) {
        in_function_body = 0;
        printf "\n";
    }
}

# Comment line in function body encountered, so strip the leading '#' and print it.
/^[ ]*\# / {
    if (in_function_body == 1) {
        sub(/^[ ]*\# /, "    ", $0);
        print $0;
    }
}

# End of function body encountered, so reset state.
/^\}$/ {
    if (in_function_body == 1) {
        in_function_body = 0;
        printf "\n";
    }
}
EOF
)
readonly function_docs_awk

print_function_docs() {
    # Print all public functions and their documentation.
    awk "${function_docs_awk}" "${THIS}"
}

config_vars_awk=$(cat <<'EOF'
BEGIN {
    comment_acc = "";
    in_env_section = 0;
}

# Comment encountered. If we are in the env section, append the line to
# the 'comment_acc' without the leading '#'.
/^# [a-zA-Z0-9\-_]/ {
    if (in_env_section == 1) {
        sub(/^# /, "    ", $0);
        # Yeah, following is string concatenation in awk...
        comment_acc = comment_acc $0 "\n";
    }
}

# Special beginning marker encountered, accumulation can start.
$0 == "# --- BEGIN ENV VARS ---" {
    in_env_section = 1;
}

# Special ending marker encountered, no point in continuing further.
$0 == "# --- END ENV VARS ---" {
    exit 0;
}

# Variable assignment encountered. Print its name and all preceding comment
# lines accumulated. Reset the accumulator.
/^RPIZFS_[a-zA-Z0-9-_]+\=/ {
    sub(/\=.*$/, "", $0);
    print $0;
    print comment_acc;
    comment_acc = "";
}
EOF
)
readonly config_vars_awk

print_config_vars_docs() {
    # Print all 'RPIZFS_' variables and their documentation that may be configured
    # by the user.
    awk "${config_vars_awk}" "${THIS}"
}

print_config_values() {
    # Print the values of all 'RPIZFS_' variables that be configured by the user.
    # They are in shell format - "name=value" without spaces - so can be dumped
    # to a file and then sourced from it again.
    for varname in $(grep -E -o -e '^RPIZFS_\w+' "${THIS}"); do
        echo -n "$varname="
        eval echo "\$${varname}"
    done
}

check_system() {
    # Check necessary preconditions of the installation, like
    # existence of the disk, permissions, etc.
    echo -n "Checking existence of '${rpizfs_imgpath}' ... "
    if test -f "${rpizfs_imgpath}"; then
        echo "OK"
    else
        echo "will need to be downloaded."
    fi

    echo -n "Checking existence of '${RPIZFS_DISK}' disk ... "
    if test -e "${rpizfs_diskpath}"; then
        echo "OK"
    else
        echo "does not exist! Exiting."
        exit 1
    fi

    echo -n "Checking ability to write to '${rpizfs_mntdir}' directory ... "
    if test -w "${rpizfs_mntdir}"; then
        echo "OK"
    else
        echo "not writable! Exiting."
        exit 1
    fi
}

_fetch_image() {
    # Fetch the installation image. No-op if the image is already present.
    if test -f "${rpizfs_imgpath}"; then
        echo "Image already downloaded. Skipping"
    else
        local remote_image_uri="${RPIZFS_REMOTE_URL}/${RPIZFS_IMG}"
        fetch "${remote_image_uri}" -o "${rpizfs_imgpath}"
    fi
}

_dd_to_disk() {
    # Burn the FreeBSD install image to the disk.
    dd \
        if="${rpizfs_imgpath}" \
        of="${rpizfs_diskpath}" \
        bs=4M \
        iflag=direct \
        oflag=direct \
        status=progress
}

_resize_partitions() {
    # Resize and modified partitions created by the '_dd_to_disk' function.
    gpart resize -i 2 "${RPIZFS_DISK}"
    gpart modify -i 1 "${rpizfs_partition}"
    gpart resize \
        -i 1 \
        -a 1M \
        "${rpizfs_partition}"
}

_create_rootzfs() {
    # Create a ZFS pool and the filesystem hierarchy within it.
    :
    # https://openzfs.github.io/openzfs-docs/man/master/8/zpool-create.8.html
    zpool create \
        -f \
        -O compress=lz4 \
        -O atime=off \
        -o ashift=12 \
        -o altroot="${rpizfs_altroot}" \
        -m none \
        "${RPIZFS_POOLNAME}" \
        "${rpizfs_partition}"

    # https://openzfs.github.io/openzfs-docs/man/master/8/zfs-create.8.html
    zfs create -o mountpoint=none "${RPIZFS_POOLNAME}/ROOT"
    zfs create -o mountpoint=/ "${RPIZFS_POOLNAME}/ROOT/default"
    zfs create -o mountpoint=/tmp -o exec=on -o setuid=off "${RPIZFS_POOLNAME}/tmp"
    zfs create -o mountpoint=/usr -o canmount=off "${RPIZFS_POOLNAME}/usr"
    zfs create -o setuid=off "${RPIZFS_POOLNAME}/usr/ports"
    zfs create "${RPIZFS_POOLNAME}/usr/src"
    zfs create -o mountpoint=/var -o canmount=off "${RPIZFS_POOLNAME}/var"
    zfs create -o exec=off -o setuid=off "${RPIZFS_POOLNAME}/var/audit"
    zfs create -o exec=off -o setuid=off "${RPIZFS_POOLNAME}/var/crash"
    zfs create -o exec=off -o setuid=off "${RPIZFS_POOLNAME}/var/log"
    zfs create -o atime=on "${RPIZFS_POOLNAME}/var/mail"
    zfs create -o setuid=off "${RPIZFS_POOLNAME}/var/tmp"

    zfs set mountpoint="${RPIZFS_POOLMOUNTPOINT}" "${RPIZFS_POOLNAME}"
    zpool set bootfs="${RPIZFS_POOLNAME}/ROOT/default" "${RPIZFS_POOLNAME}"
    zfs set canmount=noauto "${RPIZFS_POOLNAME}/ROOT/default"
}

_copy_image_to_rootzfs() {
    # Create a virtual disk from the install image and copy it to the filesystem
    # created by the '_create_rootzfs' function.
    local md_disk_name="md123";                        readonly md_disk_name
    local md_partition_disk="/dev/${md_disk_name}s2a"; readonly md_partition_disk
    local mnt_dir="${rpizfs_mntdir}/rpi-rootzfs";      readonly mnt_dir

    # Create a "virtual disk" from the install image.
    # https://man.freebsd.org/cgi/man.cgi?mdconfig
    mdconfig -a -t vnode -f "${rpizfs_imgpath}" -u "${md_disk_name}"

    mkdir -p "${mnt_dir}"
    mount "${md_partition_disk}" "${mnt_dir}"

    # The critical step - the mounted install image is archived in its entirety,
    # and then unarchived in the ZFS file system hierarchy created by the
    # '_create_rootzfs' function.
    ( cd "${mnt_dir}" && tar cf - . ) | ( cd "${rpizfs_altroot}" && tar xf - )

    # Clean-up: Unmount the virtual disk, destroy it and remove the mount
    #   directory.
    umount "${mnt_dir}"
    mdconfig -d -u "${md_disk_name}"
    rm -rf "${mnt_dir}"
}

_adjust_boot_conf() {
    # Adjust configuration so the newly installed Raspberry Pi boots correctly
    # on ZFS.
    :
    # https://man.freebsd.org/cgi/man.cgi?loader.conf
    echo 'zfs_load="YES"' >> "${rpizfs_altroot}/boot/loader.conf"
    # Following allows time synchronization running as a non-root user.
    # https://man.freebsd.org/cgi/man.cgi?query=mac_ntpd&sektion=4&format=html
    echo 'mac_ntpd_load="YES"' >> "${rpizfs_altroot}/boot/loader.conf"
    # I enable Wireguard kernel module by default.
    # https://man.freebsd.org/cgi/man.cgi?query=if_wg&sektion=4&format=html
    echo 'if_wg_load="YES"' >> "${rpizfs_altroot}/boot/loader.conf"
    # Comment out mounting UFS file system.
    sed -i '' '/ufs/ s/^/#/' "${rpizfs_altroot}/etc/fstab"
    # Comment out mounting tmpfs on '/tmp' directory. There is a separate ZFS
    # dataset for it.
    sed -i '' '/tmpfs/ s/^/#/' "${rpizfs_altroot}/etc/fstab"
}

_create_rc_conf() {
    # Create a basic '/etc/rc.conf' for the newly installed Raspberry Pi.
    :
    # https://man.freebsd.org/cgi/man.cgi?rc.conf
    echo 'hostname="rpi-rootzfs"' > "${rpizfs_altroot}/etc/rc.conf"
    echo 'sshd_enable="YES"' >> "${rpizfs_altroot}/etc/rc.conf"
    echo 'sendmail_enable="NONE"' >> "${rpizfs_altroot}/etc/rc.conf"
    echo 'ifconfig_DEFAULT="DHCP"' >> "${rpizfs_altroot}/etc/rc.conf"
    # https://docs.freebsd.org/en/books/handbook/config/#configtuning-syslog
    # That's the default, but have it here anyway.
    echo 'syslogd_flags="-s"' >> "${rpizfs_altroot}/etc/rc.conf"

    # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
    echo 'dumpdev="NO"' >> "${rpizfs_altroot}/etc/rc.conf"
    echo 'powerd_enable="YES"' >> "${rpizfs_altroot}/etc/rc.conf"
    echo 'zfs_enable="YES"' >> "${rpizfs_altroot}/etc/rc.conf"

    # If the 'mac_ntpd_load' is loaded, this enables running the 'ntpd' daemon
    # as a non-root user and synchronizes time on startup.
    echo 'ntpd_enable="YES"' >> "${rpizfs_altroot}/etc/rc.conf"
    echo 'ntpd_sync_on_start="YES"' >> "${rpizfs_altroot}/etc/rc.conf"
    echo 'ntpd_user="ntpd"' >> "${rpizfs_altroot}/etc/rc.conf"
}

_export_zpool() {
    # "Unmount" the ZFS pool created.
    :
    # https://openzfs.github.io/openzfs-docs/man/master/8/zpool-export.8.html
    zpool export "${RPIZFS_POOLNAME}"
}


install() {
    # Check system and then execute the whole installation procedure.
    echo "Installation procedure initialized."
    echo "Variables used:"
    print_config_values

    echo "Checking system:"
    check_system

    # Fetch the uncompressed FreeBSD installation image ...
    _fetch_image
    # ... burn it to the external disk ...
    _dd_to_disk
    # ... tweak the disk partitions ...
    _resize_partitions
    # ... create a filesystem hierarchy in a new ZFS pool on the external disk ...
    _create_rootzfs
    # ... copy the FreeBSD installation image to the filesystem in the pool ...
    _copy_image_to_rootzfs
    # ... adjust '/boot/loader.conf' and `/etc/fstab`, so RPi boots correctly from ZFS ...
    _adjust_boot_conf
    # ... create '/etc/rc.conf' on RPi ...
    _create_rc_conf
    # ... and finally, unmount the ZFS pool.
    _export_zpool

    echo
    echo "Installation successful."
    echo "You can shutdown the RPi ('shutdown now'), remove the microSD card and start it again."
}

# "$0" Dispatch pattern
# https://www.oilshell.org/blog/2021/07/blog-backlog-1.html#shell-programming-patterns
"$@"

  1. If “trick” and “shell script” used in one sentence together upset you, I understand. Yet, in some way, this power at the tip of your fingers is so mesmerizing, so tempting…↩︎