Samba in a FreeBSD 14 VNET jail

Recently, I set up a Samba server on Raspberry Pi 4B at home. Its use case being a place for me and my wife to store (from several devices and OSes) stuff as well as to share it among ourselves without sending it over the globe via every BigTech server imaginable.

I chose FreeBSD as the base operating system, since I wanted to explore its built-in support for OS-level containerization - jails - and its file system - ZFS.

Disclaimer: I am just a guy on the Internet. What works for me may not work for thee. I encourage you do your own research and testing.

What is a FreeBSD VNET jail?

Jails are FreeBSD’s built-in tool for running services in a contained manner. In short, you take chroot(8), add isolation of system processes, resources and users and you get jails.

VNET jails allow you to separate the jail’s whole network stack from the host on top of that, meaning it has its own network interfaces, IP address, routing table, firewall rules, …

I find VNET jail to be a good fit for running Samba, since I do not want to pollute the host with additional packages, expose too many ports on it (which Samba does) and limit the blast radius were the service compromised.

Constraints I impose on the system

If you want to have a file server to which you can connect from Linux, FreeBSD, macOS, iOS, Android, Windows, … without any major compatibility and installation hassles, Samba is pretty much the only option. Easy.

Second point means having 2 disks connected to the RPi in a mirrored setup. I decided to spend 1 innovation token here and go for ZFS as the file system and volume manager. It supports mirrored setup and dataset snapshotting out of the box and offers flexible file system managment.

Partitioning SSDs

As you can see, I have 2 SSDs connected to the RPi via USB-SATA adapters. I cannot use whole disks for the pool, because they are of slightly different sizes (both advertised as 1 “TB” by the manufacturers though). Here is an excerpt from the information provided by geom(8):

root@on-host# geom disk list
...
Geom name: da0
Providers:
1. Name: da0
   Mediasize: 1000204886016 (932G)
   Sectorsize: 512
   Mode: r0w0e0
   ...

Geom name: da1
Providers:
1. Name: da1
   Mediasize: 1024209543168 (954G)
   Sectorsize: 512
   Mode: r0w0e0
   ...

gpart(8) to the rescue! I create a GPT partition table on both disks (named da0 and da1) and a 896GB partition, aligned to 1MB and labeled by the manufacturer and the Mediasize listed above:

root@on-host# gpart destroy -F da0
da0 destroyed
root@on-host# gpart create -s GPT da0
da0 created
root@on-host# gpart add -a 1M -s 896G -l crucial-932G -t freebsd da0
da0s1 added
root@on-host# gpart create -s GPT da1
da1 created
root@on-host# gpart add -a 1M -s 896G -l apacer-954G -t freebsd da1
da1s1 added

Setting up mirrored ZFS pool

Now is the time to create a ZFS pool. I use the labels created in the preceding step and set the ashift property, which governs the smallest IO operation on the pool and should be equal to the sector size of the disks.

It is a logarithm with base 2, but setting it to 12 results in power(2, 12) = 4KB, so it is not equal to power(2, 9) = 512 reported by geom disk list command. As discussed in the linked article, operating systems often don’t report the sector size correctly. Since the disks are relatively new, I set the ashift property to 12.

atime=off disables the system changing the access time of a file each time it’s accessed and compression=lz4 enables compression of all files, which is very much worth it - according to all the folks on the Internet.

root@on-host# zpool create -o ashift=12 storage-pool mirror gpt/crucial-932G gpt/apacer-954G
root@on-host# zfs set compression=lz4 storage-pool
root@on-host# zfs set atime=off storage-pool
root@on-host# echo 'zfs_enable="YES"' >> /etc/rc.conf

For more info about these parameters and why they might be a good idea to set this way, read the whole openzfs tuning guide.

Caution: When testing the ZFS pool, it would, after reboots, sometimes not get loaded automatically (even though the zfs_enable="YES" should take care of it) and would need a manual # zfs import ... intervention. I suspect a race condition between the kernel recognizing disks, since they go through USB, and init scripts.

Therefore I devised a following solution - modifying /etc/rc.d/zpool to wait for the da0 and da1 disks:

  #!/bin/sh
  #
  #

  # PROVIDE: zpool
  # REQUIRE: hostid disks
  # BEFORE: mountcritlocal
  # KEYWORD: nojail

  . /etc/rc.subr

  name="zpool"
  desc="Import ZPOOLs"
  rcvar="zfs_enable"
  start_cmd="zpool_start"
  required_modules="zfs"
  
+ __wait_for_disk()
+ {
+   local disk i
+   disk="$1"
+   i=0
+   while ! [ -c "$disk" ]; do
+       echo "Waiting for disk $disk..."
+       sleep 1
+       if [ "$i" -eq 5 ]; then
+           break
+       fi
+       i=$(expr "$i" + 1)
+   done
+ }
+ 
  zpool_start()
  {
    local cachefile
+ 
+   __wait_for_disk da0
+   __wait_for_disk da1
  
    for cachefile in /etc/zfs/zpool.cache /boot/zfs/zpool.cache; do
        if [ -r $cachefile ]; then

# Rest of the file...

Jails template creation

Mostly following the guide in the jails chapter in the Handbook, notable differences are:

root@on-host# zfs create -o mountpoint=/jails storage-pool/jails
root@on-host# zfs create storage-pool/jails/media
root@on-host# zfs create storage-pool/jails/containers
root@on-host# zfs create storage-pool/jails/templates

Prepare archives to extract as well as files to copy into the templates sub-directory:

root@on-host# cp /usr/freebsd-dist/base.txz /jails/media/
root@on-host# cp /etc/resolv.conf /jails/media/
root@on-host# cp /etc/localtime /jails/media/

root@on-host# zfs create storage-pool/jails/templates/14.0-RELEASE
root@on-host# tar -xf /jails/media/base.txz -C /jails/templates/14.0-RELEASE --unlink
root@on-host# cp /jails/media/resolv.conf /jails/templates/14.0-RELEASE/etc
root@on-host# cp /jails/media/localtime /jails/templates/14.0-RELEASE/etc

Update it (-b <base-dir> means chrooting into the <base-dir>, that’s why resolv.conf is necessary in the <base-dir>/etc directory):

root@on-host# freebsd-update -b /jails/templates/14.0-RELEASE fetch install
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching metadata signature for 14.0-RELEASE from update1.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
Preparing to download files... done.
The following files will be updated as part of updating to
14.0-RELEASE-p5:
... # cut for brevity
Installing updates...Scanning /jails/templates/14.0-RELEASE/usr/share/certs/untrusted for certificates...
Scanning /jails/templates/14.0-RELEASE/usr/share/certs/trusted for certificates...
 done.

root@on-host# zfs snapshot storage-pool/jails/templates/14.0-RELEASE@base

Installing Samba and other provisioning

root@on-host# zfs clone storage-pool/jails/templates/14.0-RELEASE@base storage-pool/jails/containers/samba

root@on-host# pkg -c /jails/containers/samba/ install samba416-4.16.11_2

I use the -c <chroot-dir> in the pkg command to chroot into the /jails/containers/samba, therefore Samba is actually installed in that directory and invoking smbclient on the host results in an error:

root@on-host# smbclient --version
su: smbclient: not found

root@on-host# chroot /jails/containers/samba/ smbclient --version
Version 4.16.11

That is something I really like about FreeBSD - both freebsd-update and pkg integrate with chroot/jails seamlessly.

Here are the contents of the smb4.conf that need to be put into /usr/local/etc:

[global]
workgroup = WORKGROUP
server string = SambaHome
server role = standalone server
hosts allow = 192.168.120.
netbios name = SambaHome
wins support = no
security = user
passdb backend = tdbsam
encrypt passwords = yes

[libor]
path = /opt/libor
valid users = @owners
write list = libor
writable  = yes
browsable = yes
guest ok = no
create mask = 0666
directory mask = 0755

[shared]
path = /opt/shared
valid users = guest @owners
write list = guest @owners
writable  = yes
browsable = yes
guest ok = yes
guest user = guest
create mask = 0666
directory mask = 0775

Note that all steps may be carried out from the host system. When dealing with users, we just need to issue commands prepended by chroot.

I decided (arbitrarily) to place the Samba root in the /opt directory, so I create a dataset:

root@on-host# zfs create storage-pool/jails/containers/samba/opt

and, since it will contain mostly larger files like photos, I set the recordsize dataset attribute to 1MB instead of the default 128KB. You can check the following article for more details about setting the right recordsize.

root@on-host# zfs set recordsize=1M storage-pool/jails/containers/samba/opt

root@on-host# chroot /jails/containers/samba pw group add owners

Adding libor user (guest is a copy-pasta, it just does not belong to the owners group):

root@on-host# chroot /jails/containers/samba adduser
Username: libor
Full name: Libor
Uid (Leave empty for default): 
Login group [libor]: 
Login group is libor. Invite libor into other groups? []: owners
Login class [default]: 
Shell (sh csh tcsh nologin) [sh]: nologin
Home directory [/home/libor]: /nonexistent
Home directory permissions (Leave empty for default): 
Use password-based authentication? [yes]: no
Lock out the account after creation? [no]: no
Username   : libor
Password   : <disabled>
Full Name  : Libor
Uid        : 1001
Class      : 
Groups     : libor owners
Home       : /nonexistent
Home Mode  : 
Shell      : /usr/sbin/nologin
Locked     : no
OK? (yes/no) [yes]: yes
adduser: INFO: Successfully added (libor) to the user database.
Add another user? (yes/no) [no]: yes
... # dialog starts over again...

root@on-host# chroot /jails/containers/samba mkdir /opt/libor /opt/shared
root@on-host# chroot /jails/containers/samba chown -R libor:owners /opt/libor
root@on-host# chroot /jails/containers/samba chown -R guest:owners /opt/shared

root@on-host# chroot /jails/containers/samba smbpasswd -a libor
New SMB password:
Retype new SMB password:
Added user libor.

I copy the /jails/media/samba/rc.conf to /jails/containers/samba/etc. It’s a minimal init script, with auxiliary services disabled (sshd, syslogd, sendmail, cron).

sshd_enable="NO"
sendmail_enable="NONE"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
growfs_enable="NO"

syslogd_enable="NO"
cron_enable="NO"

samba_server_enable="YES"

Lastly, I create a snapshot the storage-pool/jails/containers/samba dataset to save myself some trouble in case I botch something in the future:

root@on-host# zfs snapshot storage-pool/jails/containers/samba@base

Starting the jail

Almost everything is now (finally!) in place to start the jail. This follows closely VNET section in the Handbook, so if details are unclear, refer to it.

We first need to create a network bridge device and attach it to the network interface of the RPi. Since I want it to be a permanent change, I put the commands into /etc/rc.conf directly:

root@on-host# cat <<EOF >> /etc/rc.conf

defaultrouter="192.168.120.1"
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 192.168.120.120/24 addm genet0 up"
EOF

root@on-host# cat <<EOF >> /etc/rc.conf

jail_enable="YES"
jail_parallel_start="YES"
EOF

root@on-host# echo '.include "/etc/jail.conf.d/*.conf";' >> /etc/jail.conf

And create the following /etc/jail.conf.d/samba.conf file, conforming to the jail.conf(5) format:

samba {
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.consolelog = "/var/log/jail_console_${name}.log";

  enforce_statfs = "1";
  allow.raw_sockets;
  exec.clean;
  allow.mount;
  allow.mount.devfs = 1;
  allow.mount.fdescfs;
  mount.devfs;
  devfs_ruleset = 5;

  path = "/jails/containers/${name}";
  host.hostname = "${name}";

  vnet;
  vnet.interface = "${epair}b";

  $id = "123";
  $ip = "192.168.120.${id}/24";
  $gateway = "192.168.120.1";
  $bridge = "bridge0";
  $epair = "epair${id}";

  exec.prestart += "ifconfig ${epair} create up";
  exec.prestart += "ifconfig ${epair}a up descr jail:${name}";
  exec.prestart += "ifconfig ${bridge} addm ${epair}a up";
  exec.start    += "ifconfig ${epair}b ${ip} up";
  exec.start    += "route add default ${gateway}";
  exec.poststop = "ifconfig ${bridge} deletem ${epair}a";
  exec.poststop += "ifconfig ${epair}a destroy";
}

What’s going on here?

With this file, I declare I want a VNET jail with the name samba, whose base directory (chroot) will be /jails/containers/samba, it will have a static IP address 192.168.120.123 and all its logs will appear in /var/log/jail_console_samba.log on the host (lines: 18, 1, 15 and 22, 4).

Said jail communicates with the host via an epair(4), which is a virtual Ethernet connection, with one end - epair123a - residing in the host network stack and the other - epair123b - in the jail network stack, connecting them this way.

Setup before jail start (on the host) is specified on lines 27-29, after the jail start (within the jail!) on lines 30-31 and after the jail stop (on the host) on lines 32-33. They are concerned with creation/destruction of the epair, connecting to/disconnecting from the bridge and setting the default route in the jail.

Lines 6-13 are what’s needed to run Samba successfully in a jailed environment, otherwise it complains about missing privileges for mounting and other stuff.

Frankly, it’s bit unclear to me what all the options mean and whether they are really necessary, so I prescribe myself a homework to figure it out. My assumption is something about Samba running the rpcbind(8) under the hood, therefore maybe needing raw sockets (?) and mounting file systems, therefore needing access to the host’s /dev filesystem, but I dunno.

root@on-host# service jail start samba
Starting jails: samba.

root@on-host# cat /var/log/jail_console_samba.log
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/samba4
32-bit compatibility ldconfig path: /usr/lib32
Starting Network: lo0 epair123b.
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair123b: flags=1008842<BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:fb:c0:41:5a:0b
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
add host 127.0.0.1: gateway lo0 fib 0: route already in table
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Clearing /tmp (X related).
Updating motd:.
Updating /var/run/os-release done.
Creating and/or trimming log files.
Performing sanity check on Samba configuration: OK
Starting nmbd.
Starting smbd.

Tue Apr  9 22:41:26 CEST 2024
add net default: gateway 192.168.120.1

You can see the epair123b end of the epair as the primary interface, smbd/nmbd starting, samba config is actually OK, etc…

However, we cannot see the IP address (yet), so I issue ifconfig(8) with -j <jail_name> option, meaning it will be executed in the supplied jail:

root@on-host# ifconfig -j samba
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair123b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:9f:23:30:dc:0b
    inet 192.168.120.123 netmask 0xffffff00 broadcast 192.168.120.255
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Now the jail has an IP address we assigned it - 192.168.120.123, as the epair123b went from DOWN to UP.

root@on-host# sockstat -j samba -l -w  # again, `-j <jail_name>` executes the command in the jail
USER     COMMAND    PID   FD  PROTO  LOCAL ADDRESS                                 FOREIGN ADDRESS
root     smbd        2579 5   dgram  /var/db/samba4/private/msg.sock/2579
root     smbd        2578 5   dgram  /var/db/samba4/private/msg.sock/2578
root     smbd        2555 5   dgram  /var/db/samba4/private/msg.sock/2555
root     smbd        2555 28  tcp6   *:445                                         *:*
root     smbd        2555 29  tcp6   *:139                                         *:*
root     smbd        2555 30  tcp4   *:445                                         *:*
root     smbd        2555 31  tcp4   *:139                                         *:*
root     nmbd        2551 5   dgram  /var/db/samba4/private/msg.sock/2551
root     nmbd        2551 13  udp4   *:137                                         *:*
root     nmbd        2551 14  udp4   *:138                                         *:*
root     nmbd        2551 15  udp4   192.168.120.123:137                           *:*
root     nmbd        2551 16  udp4   192.168.120.255:137                           *:*
root     nmbd        2551 17  udp4   192.168.120.123:138                           *:*
root     nmbd        2551 18  udp4   192.168.120.255:138                           *:*
root     nmbd        2551 19  stream /var/run/samba4/nmbd/unexpected

I don’t see anything unexpected. Ports 137, 138, 139 and 445 are all Samba-related.

Testing Samba

When I navigate in my laptop’s file explorer to smb://192.168.120.123, I see 2 directories (Samba shares) - libor and shared.

Trying to enter the libor directory results in a password prompt. guest user, even with correct password, is rejected, even though shared directory is accesible by them.

Of course, it’s just me uploading 1 (!) big file, so performance may be wildly different on multiple smaller files, but I think for home use, it is more than sufficient.

Final words and future plans

I enjoyed this exercise really, really much. Sure, there were some hiccups (patching the /etc/rc.d/zpool…), but overall, FreeBSD is such a pleasant, coherent operating system that I will rely on more in the future and build more services atop of.

Also, ZFS innovation token seems to pay off - it sure is daunting to get started with, but its flexibility and the fact that so many features are baked in makes me really curious about it and I want to dig deeper into its functionality.

The goal post of potential improvements is always moving, that’s a universal law. So, I came up with these enhancements:

To be even more sure about the durability of the data stored on the disks, I would like to create backups that are geographically separated from my closet.

To achieve that, I would like to set up a WireGuard virtual private network, connect the file server and other machines to it and send snapshots to a dedicated machine via zfs-send.8.

Big thanks go to my wife for providing stylistic guidance as well as picture edits.

Setting up Samba server in a FreeBSD 14 VNET jail