Building a BSD home router (pt. 8): ZFS and jails

Previous parts of this series:

Part 1 (discussing why you want to build your own router and how to assemble the APU2),
Part 2 (some Unix history explanation of what a serial console is),
Part 3 (demonstrating serial access to the APU and covering firmware update),
Part 4 (installing pfSense),
Part 5 (installing OPNsense instead)
Part 6 (Comparison of pfSense and OPNsense)
Part 7 (Advanced installation of OPNsense)

Fixing swap

This is the last part of this series of building a BSD home router. In the previous article we did an advanced setup of OPNsense that works but is currently wasting valuable disk space. We also configured OPNsense for SSH access. Now let’s SSH in and su – to root and continue! Choose shell (menu point 8) so that we can have a look around.

# df -h
Filesystem           Size    Used   Avail Capacity  Mounted on
/dev/ufs/OPNsense    1.9G    909M    916M    50%    /
devfs                1.0K    1.0K      0B   100%    /dev
/dev/ada0s1b         991M    8.0K    912M     0%    /none
devfs                1.0K    1.0K      0B   100%    /var/dhcpd/dev

Uhm… ada0s1b is mounted on /none? Seriously? Let’s get rid of that real quick:

# umount /none

How did that happen? This leads to the question: What does our disklabel on slice 1 look like?

# gpart show ada0s1
=>      0  6290865  ada0s1  BSD  (3.0G)
        0       16          - free -  (8.0K)
       16  4194288       1  freebsd-ufs  (2.0G)
  4194304  2096561       2  freebsd-ufs  (1.0G)

There you have it. The second one is all wrong, it’s not meant to be UFS! We have to correct it to have proper swap space configured:

# gpart delete -i 2 ada0s1
ada0s1b deleted
# gpart add -t freebsd-swap ada0s1
ada0s1b added
# swapon /dev/ada0s1b
# swapinfo 
Device          1K-blocks     Used    Avail Capacity
/dev/ada0s1b      1048280        0  1048280     0%

That’s better. Now we need to adjust fstab to make this change persistent:

# vi /etc/fstab

Change the ada0s1b line like this:

/dev/ada0s1b		none		swap	sw		0	0

Ok, we have some swap now, but we’re wasting most of the disk space of our drive. Let’s address that one next!

Preparing the system for ZFS

In the installer we created a second slice (MBR partition) as a placeholder:

# gpart show ada0
=>      63  31277169  ada0  MBR  (15G)
        63   6290865     1  freebsd  [active]  (3.0G)
   6290928  24986304     2  !57  (12G)

Let’s delete it and create a second FreeBSD slice instead:

# gpart delete -i 2 ada0
ada0s2 deleted
# gpart add -t freebsd ada0
ada0s2 added

Now we need to create a disklabel inside and create a partition for ZFS:

# gpart create -s bsd ada0s2
ada0s2 created
# gpart add -t freebsd-zfs ada0s2
ada0s2a added

OPNsense does not load the ZFS kernel module by default. So let’s do that now and also notify the loader to always insert that ko during startup (we’re using loader.conf.local because OPNsense overwrites loader.conf during startup):

# kldload zfs
# echo zfs_load=\"YES\" >> /boot/loader.conf.local

Then we set the ashift. This tells ZFS to adjust to a 4k blocksize which is better for most of today’s drives use instead of 512 byte ones, even though a lot of them will lie to you and claim to have 512 byte sector size. But even on a drive that really has 512 byte sectors, using 4k is better than using 512 bytes on a 4k sector drive. You will only lose some space if you have a lot of very small files in this case. In the other case however, you will hurt performance badly. If you know your drive and you want to use another blocksize, look up how to do it. Otherwise just set the ashift like this:

# sysctl vfs.zfs.min_auto_ashift=12
vfs.zfs.min_auto_ashift: 9 -> 12

With that we’re good to go and create a pool and some datasets.

Pool creation

I’m calling my pool zdata but feel free to name yours whatever you like better. I also enable compression on the pool level and turn off atime:

zpool create -O compression=lz4 -O atime=off -O mountpoint=none zdata /dev/ada0s2a

Next is creating some basic datasets that won’t be used directly (hence forbidden to mount) but only serve as parents for other datasets:

# zfs create -o canmount=off -o mountpoint=none zdata/var
# zfs create -o canmount=off -o mountpoint=none zdata/usr

Let’s move the old log dir and create some new directories:

# mv /var/log /var/log.old
# mkdir /var/log
# mkdir /usr/ports

On with some more datasets:

# zfs create -o mountpoint=legacy zdata/var/log
# zfs create -o mountpoint=legacy zdata/usr/ports
# zfs create -o mountpoint=legacy zdata/usr/obj

To make the system use those we need to add them to the fstab:

# vi /etc/fstab

Add these lines to the file:

zdata/var/log		/var/log	zfs	rw		0	0
zdata/usr/ports		/usr/ports	zfs	rw		0	0
zdata/usr/obj		/usr/obj	zfs	rw		0	0

Once these additional lines are in place, the datasets can be mounted and the old logs transferred to their new place:

# mount -a
# mv /var/log.old/* /var/log/

The directory /var/log.old is no longer needed, but the system currently has some file descriptors open that prevent deleting it. Just rmdir after the next reboot. Speaking of which: It is now a good time to do updates (and change the firmware to the libressl-based one if you haven’t switched already).

BTW: Don’t try to put everything on ZFS! I made some experiments booting into single user mode and moving over /usr and /var. The results were… not pleasing. After doing some reading I found that while OPNsense works well with ZFS datasets, it’s startup process doesn’t cope with ZFS very well. Place its configuration on ZFS and you’re left with a partially defunct system (that doesn’t know its hostname and won’t start a lot of things that are needed).

Full ZFS support is already on the wish list for OPNsense. It looks like that won’t make it into 17.7, but I’m pretty sure that it will eventually be available, making root-on-ZFS installations possible. Yes, pfSense already has that feature in their betas for the upcoming version 2.4. And they even ditched the DragonFly installer and use the familiar BSDinstall which is really cool (dear OPNsense devs, please also take this step in the future, it would be greatly appreciated!).

Is this a good reason to switch to pfSense? It might, if for you this is the one killer feature and you are willing to let go of OPNsense’s many improvements. But there’s one big blocker: If you make the switch you don’t really need to read on. You won’t be able to create jails easily. Why? Because pfSense heavily customizes FreeBSD. So heavily in fact that you cannot even use the ports tree by default! And that is truly a rather sad state of affairs. Sure, a lot of pfSense users actually use MacOS or even Windows and only want to ever interact with the GUI. BSD means nothing to them at all. But if you’re a FreeBSD user it’s pretty annoying if things simply don’t work (and OPNsense shows that there’s no real need to screw things up as much as pfSense does it).

Ports and jails

The OPNsense team provides packages for OPNsense that you can simply install via pkg. However they currently offer only 368 packages, so chances are that you want something that is not there. The FreeBSD ports tree on the other hand means that over 27,000 programs are easily available for you! So since OPNsense is based on FreeBSD (and tries to remain close to it) this is really an option.

On FreeBSD you’d probably use portsnap to get a snapshot of the current ports tree. This won’t work in our case since OPNsense doesn’t have that tool. The other common way on FreeBSD is to use svnlite and checkout the ports tree from the Subversion repo. Again OPNsense doesn’t provide that tool. And it also doesn’t package the full SVN.

So what can we do to acquire the ports tree? OPNsense does provide a git package and the FreeBSD project offers a git mirror of the SVN repositories. But wait a second! OPNsense works together with the HardenedBSD team and they have their own ports tree (based on the vanilla FreeBSD one with some additions). The whole ports tree is pretty big, but we don’t really want (or need) the whole history. Just what various version control systems call “head”, “tip”, “leaf”, … For git we can achieve this setting the “depth” to 1:

# pkg install git
# git clone --depth=1 https://github.com/HardenedBSD/hardenedbsd-ports.git /usr/ports

FreeBSD ships with OpenSSL in base and a lot of ports expect to link against that. We’re however using LibreSSL and so we have to tell the build system to use that by making an entry in make.conf:

# echo DEFAULT_VERSIONS+=ssl=libressl >> /etc/make.conf

If – for whatever reason – you decided to stick to the OpenSSL firmware, you still need to edit make.conf. This is because OPNsense uses OpenSSL from ports which is usually newer than the version from base (that cannot be upgraded between releases for ABI stability reasons). Use ssl=openssl in that case.

The next step is optional, but I recommend installing a tool for dealing with ports. My example is a pretty light-weight port but maybe you want to build something more demanding. Especially in those cases a ports management tool comes in very handy. I suggest portmaster which is extremely light-weight itself:

# make -C /usr/ports/ports-mgmt/portmaster install clean

Once you have it installed, you can install the jail management tool. Yes, I know that I’ve written about py3-iocage a while ago, but that comes with a lot of dependencies and doesn’t provide enough of an advantage over the purely shell based iocell fork. For that reason I would simply go with that one in this case:

# portmaster sysutils/iocell

Alright! Now you have iocage installed and can start creating jails. What services would you want to jail on a small router box that is always on? Think about it for a moment. There are many great possibilities (I’ll likely write another article soon about what I have in mind right now).

Looking back – and forward

What have we accomplished in this series? I now have a frugal little router on my desk that is quietly doing its work. So far it’s just an additional machine between my network and the modem/router box from my ISP. Taking a break from topics directly related to the actual router, I’ll setup some jails (and NAT) next. But then there is a lot more to look into: How to do proper firewalling? What about traffic shaping? How to configure logging? Also VPN and VoIP come to mind as well as NTP, a DNS cache or even vLANs or intrusion detection.

OPNsense places so many tools within reach of your hands. You only have to grab one of them at a time and learn to use it. That’s what I intend to do. And then, some point in the future, equipped with much more solid networking knowledge, I’ll try to replace that box I got from my ISP with my own modem, too. But excuse me now, I have some reading to do and configurations to break and fix again.