Building a BSD home router (pt. 8): ZFS and jails

Previous parts of this series:

Part 1 (discussing why you want to build your own router and how to assemble the APU2),
Part 2 (some Unix history explanation of what a serial console is),
Part 3 (demonstrating serial access to the APU and covering firmware update),
Part 4 (installing pfSense),
Part 5 (installing OPNsense instead)
Part 6 (Comparison of pfSense and OPNsense)
Part 7 (Advanced installation of OPNsense)

Fixing swap

This is the last part of this series of building a BSD home router. In the previous article we did an advanced setup of OPNsense that works but is currently wasting valuable disk space. We also configured OPNsense for SSH access. Now let’s SSH in and su – to root and continue! Choose shell (menu point 8) so that we can have a look around.

# df -h
Filesystem           Size    Used   Avail Capacity  Mounted on
/dev/ufs/OPNsense    1.9G    909M    916M    50%    /
devfs                1.0K    1.0K      0B   100%    /dev
/dev/ada0s1b         991M    8.0K    912M     0%    /none
devfs                1.0K    1.0K      0B   100%    /var/dhcpd/dev

Uhm… ada0s1b is mounted on /none? Seriously? Let’s get rid of that real quick:

# umount /none

How did that happen? This leads to the question: What does our disklabel on slice 1 look like?

# gpart show ada0s1
=>      0  6290865  ada0s1  BSD  (3.0G)
        0       16          - free -  (8.0K)
       16  4194288       1  freebsd-ufs  (2.0G)
  4194304  2096561       2  freebsd-ufs  (1.0G)

There you have it. The second one is all wrong, it’s not meant to be UFS! We have to correct it to have proper swap space configured:

# gpart delete -i 2 ada0s1
ada0s1b deleted
# gpart add -t freebsd-swap ada0s1
ada0s1b added
# swapon /dev/ada0s1b
# swapinfo 
Device          1K-blocks     Used    Avail Capacity
/dev/ada0s1b      1048280        0  1048280     0%

That’s better. Now we need to adjust fstab to make this change persistent:

# vi /etc/fstab

Change the ada0s1b line like this:

/dev/ada0s1b		none		swap	sw		0	0

Ok, we have some swap now, but we’re wasting most of the disk space of our drive. Let’s address that one next!

Preparing the system for ZFS

In the installer we created a second slice (MBR partition) as a placeholder:

# gpart show ada0
=>      63  31277169  ada0  MBR  (15G)
        63   6290865     1  freebsd  [active]  (3.0G)
   6290928  24986304     2  !57  (12G)

Let’s delete it and create a second FreeBSD slice instead:

# gpart delete -i 2 ada0
ada0s2 deleted
# gpart add -t freebsd ada0
ada0s2 added

Now we need to create a disklabel inside and create a partition for ZFS:

# gpart create -s bsd ada0s2
ada0s2 created
# gpart add -t freebsd-zfs ada0s2
ada0s2a added

OPNsense does not load the ZFS kernel module by default. So let’s do that now and also notify the loader to always insert that ko during startup (we’re using loader.conf.local because OPNsense overwrites loader.conf during startup):

# kldload zfs
# echo zfs_load=\"YES\" >> /boot/loader.conf.local

Then we set the ashift. This tells ZFS to adjust to a 4k blocksize which is better for most of today’s drives use instead of 512 byte ones, even though a lot of them will lie to you and claim to have 512 byte sector size. But even on a drive that really has 512 byte sectors, using 4k is better than using 512 bytes on a 4k sector drive. You will only lose some space if you have a lot of very small files in this case. In the other case however, you will hurt performance badly. If you know your drive and you want to use another blocksize, look up how to do it. Otherwise just set the ashift like this:

# sysctl vfs.zfs.min_auto_ashift=12
vfs.zfs.min_auto_ashift: 9 -> 12

With that we’re good to go and create a pool and some datasets.

Pool creation

I’m calling my pool zdata but feel free to name yours whatever you like better. I also enable compression on the pool level and turn off atime:

zpool create -O compression=lz4 -O atime=off -O mountpoint=none zdata /dev/ada0s2a

Next is creating some basic datasets that won’t be used directly (hence forbidden to mount) but only serve as parents for other datasets:

# zfs create -o canmount=off -o mountpoint=none zdata/var
# zfs create -o canmount=off -o mountpoint=none zdata/usr

Let’s move the old log dir and create some new directories:

# mv /var/log /var/log.old
# mkdir /var/log
# mkdir /usr/ports

On with some more datasets:

# zfs create -o mountpoint=legacy zdata/var/log
# zfs create -o mountpoint=legacy zdata/usr/ports
# zfs create -o mountpoint=legacy zdata/usr/obj

To make the system use those we need to add them to the fstab:

# vi /etc/fstab

Add these lines to the file:

zdata/var/log		/var/log	zfs	rw		0	0
zdata/usr/ports		/usr/ports	zfs	rw		0	0
zdata/usr/obj		/usr/obj	zfs	rw		0	0

Once these additional lines are in place, the datasets can be mounted and the old logs transferred to their new place:

# mount -a
# mv /var/log.old/* /var/log/

The directory /var/log.old is no longer needed, but the system currently has some file descriptors open that prevent deleting it. Just rmdir after the next reboot. Speaking of which: It is now a good time to do updates (and change the firmware to the libressl-based one if you haven’t switched already).

BTW: Don’t try to put everything on ZFS! I made some experiments booting into single user mode and moving over /usr and /var. The results were… not pleasing. After doing some reading I found that while OPNsense works well with ZFS datasets, it’s startup process doesn’t cope with ZFS very well. Place its configuration on ZFS and you’re left with a partially defunct system (that doesn’t know its hostname and won’t start a lot of things that are needed).

Full ZFS support is already on the wish list for OPNsense. It looks like that won’t make it into 17.7, but I’m pretty sure that it will eventually be available, making root-on-ZFS installations possible. Yes, pfSense already has that feature in their betas for the upcoming version 2.4. And they even ditched the DragonFly installer and use the familiar BSDinstall which is really cool (dear OPNsense devs, please also take this step in the future, it would be greatly appreciated!).

Is this a good reason to switch to pfSense? It might, if for you this is the one killer feature and you are willing to let go of OPNsense’s many improvements. But there’s one big blocker: If you make the switch you don’t really need to read on. You won’t be able to create jails easily. Why? Because pfSense heavily customizes FreeBSD. So heavily in fact that you cannot even use the ports tree by default! And that is truly a rather sad state of affairs. Sure, a lot of pfSense users actually use MacOS or even Windows and only want to ever interact with the GUI. BSD means nothing to them at all. But if you’re a FreeBSD user it’s pretty annoying if things simply don’t work (and OPNsense shows that there’s no real need to screw things up as much as pfSense does it).

Ports and jails

The OPNsense team provides packages for OPNsense that you can simply install via pkg. However they currently offer only 368 packages, so chances are that you want something that is not there. The FreeBSD ports tree on the other hand means that over 27,000 programs are easily available for you! So since OPNsense is based on FreeBSD (and tries to remain close to it) this is really an option.

On FreeBSD you’d probably use portsnap to get a snapshot of the current ports tree. This won’t work in our case since OPNsense doesn’t have that tool. The other common way on FreeBSD is to use svnlite and checkout the ports tree from the Subversion repo. Again OPNsense doesn’t provide that tool. And it also doesn’t package the full SVN.

So what can we do to acquire the ports tree? OPNsense does provide a git package and the FreeBSD project offers a git mirror of the SVN repositories. But wait a second! OPNsense works together with the HardenedBSD team and they have their own ports tree (based on the vanilla FreeBSD one with some additions). The whole ports tree is pretty big, but we don’t really want (or need) the whole history. Just what various version control systems call “head”, “tip”, “leaf”, … For git we can achieve this setting the “depth” to 1:

# pkg install git
# git clone --depth=1 /usr/ports

FreeBSD ships with OpenSSL in base and a lot of ports expect to link against that. We’re however using LibreSSL and so we have to tell the build system to use that by making an entry in make.conf:

# echo DEFAULT_VERSIONS+=ssl=libressl >> /etc/make.conf

If – for whatever reason – you decided to stick to the OpenSSL firmware, you still need to edit make.conf. This is because OPNsense uses OpenSSL from ports which is usually newer than the version from base (that cannot be upgraded between releases for ABI stability reasons). Use ssl=openssl in that case.

The next step is optional, but I recommend installing a tool for dealing with ports. My example is a pretty light-weight port but maybe you want to build something more demanding. Especially in those cases a ports management tool comes in very handy. I suggest portmaster which is extremely light-weight itself:

# make -C /usr/ports/ports-mgmt/portmaster install clean

Once you have it installed, you can install the jail management tool. Yes, I know that I’ve written about py3-iocage a while ago, but that comes with a lot of dependencies and doesn’t provide enough of an advantage over the purely shell based iocell fork. For that reason I would simply go with that one in this case:

# portmaster sysutils/iocell

Alright! Now you have iocage installed and can start creating jails. What services would you want to jail on a small router box that is always on? Think about it for a moment. There are many great possibilities (I’ll likely write another article soon about what I have in mind right now).

Looking back – and forward

What have we accomplished in this series? I now have a frugal little router on my desk that is quietly doing its work. So far it’s just an additional machine between my network and the modem/router box from my ISP. Taking a break from topics directly related to the actual router, I’ll setup some jails (and NAT) next. But then there is a lot more to look into: How to do proper firewalling? What about traffic shaping? How to configure logging? Also VPN and VoIP come to mind as well as NTP, a DNS cache or even vLANs or intrusion detection.

OPNsense places so many tools within reach of your hands. You only have to grab one of them at a time and learn to use it. That’s what I intend to do. And then, some point in the future, equipped with much more solid networking knowledge, I’ll try to replace that box I got from my ISP with my own modem, too. But excuse me now, I have some reading to do and configurations to break and fix again.

Building a BSD home router (pt. 7): Advanced OPNsense installation

Previous parts of this series:

Part 1 (discussing why you want to build your own router and how to assemble the APU2),
Part 2 (some Unix history explanation of what a serial console is),
Part 3 (demonstrating serial access to the APU and covering firmware update),
Part 4 (installing pfSense),
Part 5 (installing OPNsense instead) and
Part 6 (Comparison of pfSense and OPNsense)

Revisiting the initial question

In the first post I asked the question “Why would you want to build your own router?” and the answer was “because the stock ones are known to totally suck”. I have since stumbled across this news: Mcafee claims: Every router in the US is compromised. Now Mcafee is a rather flamboyant personality and every is a pretty strong statement. But I’m not such a nit-picker and in general he’s definitely right. If you have a couple of minutes, read the article and/or watch the short Youtube interview that it has embedded.

If you care about things like privacy at all, we’re living in a nightmare already and things keep getting worse. What I have blogged about in this series of posts so far is not really solving any problem. It’s just a first step to take back your network. Have you built your own router, too, or are you planning to do so? Just assembling it and installing a firewall OS on it won’t do the trick. As a next step you have to learn the basics of networking and firewalling so you can configure your box according to your needs. And even then you have just put your own router behind the modem/router box from your ISP and not replaced that. I’d like to go further and get my own modem, too. But that step requires a lot more reading before I will even attempt to do it.

Manual installation

However this article is about doing a more advanced OPNsense installation that leaves room for customizing things. Let’s get to it!

OPNsense Installer: Manual installation

In the installer select “manual installation” obviously. This will lead you through a couple of dialog windows that let you customize your partitioning etc.

OPNsense Installer: Format the disk?

It seems like OPNsense can be installed on an existing filesystem. There might be people who would want that feature but I don’t. I definitely prefer to start fresh as a newly installed OS should be in a clean state in my opinion.

OPNsense installer: Geometry confirmation

The installer then gives you the option to change the disk geometry. You almost certainly don’t want to do this. If you do need to, you have a strange disk, are aware of its quirks and know geometry matters good enough that you definitely don’t need my advice on it.

OPNsense installer: Slice disk?

Next you are asked if you want to slice (OPNsense uses the term “partition” to describe MBR partitions which is fine since that’s what non-BSD people usually call it). I don’t expect to be dual-booting my box or anything, so I could go with just one slice. However I might install and try out some other versions (or take a look at pfSense again when 2.4 is officially out or even something like OpenWRT, just to take a look at it). For that reason I create two slices so I can keep my OS on one and my data on the other.

OPNsense installer: Disk slicing

I created a FreeBSD slice and one of type Plan9. No, I’m not going to put Plan9 on there. It will be erased and re-purposed anyway. But the installer has this option and Plan9 is cool. 3 GB for OPNsense should be enough and I give the rest to the future data slice.

OPNsense installer: Slice alignment

For the advanced installation we’re unfortunately stuck with installing on the MBR partitioning scheme. That means (for compatibility’s sake) the system enforces the old CHS (Cylinder, Head, Sector) addressing limitations which are almost completely irrelevant today, but meh. The most annoying consequence of this is that “partitions have to end on a cylinder boundary”. If you don’t know what that means: It’s related to the physical geometry of spinning drives that has been of high importance in the olde days(tm) and still haunt us today because operating systems are used to work with it (even though geometry parameters have been lies and lies for decades now and SSDs don’t have spinning parts but claim to have them to make the OS happy…). To comply with this, choose to grow or shrink your slice by a couple of sectors.

OPNsense installer: No bootblock installation

If you want to dual-boot (or multi-boot) your box, make sure to install a boot manager now. I don’t anticipate to install more than one OS on it at the same time and so I skip this. Oh, and please don’t ask me what “packet mode” is! I tried to research it, but all that I found boils down to “if you have problems, try with/without it”. I couldn’t really find anything about what that actually does (at least not in a reasonable amount of time). If you know: Please leave me a comment!

OPNsense installer: Slice selection

Next is selecting which slice to install to. Why, the FreeBSD one, of course!

OPNsense installer: Adding disklabel partitions

Finally the slice needs to be partitioned (or sub-partitioned if you regard the slices as “partitions”!). This means that BSD disklabels are created inside the MBR slice to allow for multiple partitions. For the setup that I have in mind, two partitions suffice: One for / and the other for SWAP space. For whatever reason the installer does not directly allow to assign SWAP, so I allocate 2 GB for the root partition and the rest to a second partition that has no mountpoint. That’s it, the installation can start now.

SSH access

Once the installation is complete, follow the steps that I wrote about in the article about the simple installation.

Got the interfaces assigned and the setup wizard run? Good. OPNsense can be administered purely through the Web GUI. Howver if you’re like me, you really prefer some means of direct console access. Sure, we have that over the serial console. While that’s fine for the installation, it’s a bit cumbersome for daily use. Fortunately there’s a better way: Let’s just enable SSH access!

OPNsense Web GUI: Creating a user

First stop: Creating a user (you wouldn’t want to SSH in as root, do you? Do you?! Do this on a production machine like never). One thing is important here: Make your new user part of the “admin” group or else it won’t be terribly useful to you. Also use SSH keys instead of passwords. If you haven’t ever used keys, set a couple of minutes aside to do a little reading about what they are. They are much more secure than passwords and you definitely want to use them even if you don’t know that just yet (I recommend the article on SSH keys over at the Arch Linux wiki. Unless you’re using the original OpenBSD OpenSSH, we’re all using the same version of OpenSSH-portable anyway). You must also check the “use scrambled password” checkbox because OPNsense won’t let you get away with an empty password.

OPNsense Web GUI: Enabling SSH

Then OpenSSH needs to be activated. If – for whatever reason – you cannot use keys, you have to enable the “permit password login” option. Try to avoid that, though. And don’t check the “permit root user login” however convenient it might be!

SSH login to the OPNsense box

That’s it, you can now log into your box using SSH. Use su – and the root PW to become root. OPNsense will then display the nice menu that you already know from connecting via serial.

What’s next?

Right now we have a lot of disk space wasted and there’s other things wrong, too. So after the installation there’s some more work to do, some packages to install, filesystems to create, etc. I originally intended to stuff more into this post but it’s certainly long enough already. See you in part 8, the last part of the series!

Building a BSD home router (pt. 6): pfSense vs. OPNsense

Part 1 of this article series was about why you want to build your own router, and how to assemble the APU2 that I chose as the hardware to build it from. Part 2 gave some Unix history and explained what a serial console is. Part 3 demonstrated serial access to the APU and showed how to update its firmware. Part 4 detailed installing pfSense, while the previous one did the same with OPNsense.

A little overview: In this post I will give you some background information, compare the appearance / usability of both products and then take a look at some special features before giving a conclusion.

pfSense vs. OPNsense: Who wins?

This article is about comparing both products and helping you to make a decision. It is not terribly in-depth, because that task would require its own series of articles (and a lot more free time for me to dig much deeper into the topic). But still there’s a lot you may want to know to get a first impression on which one you should probably choose. If you do some more research and write about it, please let me know and I will happily link to your work!

I want to point out one thing right at the beginning: Both products are good firewall solutions with a heck of a lot of extras. If you have the same goal as I have (building a home router), either will do absolutely fine. That does of course not mean that your choice doesn’t matter at all. You can definitely benefit from thinking about it before making a decision. But even making the “wrong” decision doesn’t mean that it will be horribly wrong. There are a couple of differences and maybe they are important to you. But chances are that both products would completely satisfy your needs.


Sometimes it’s helpful to ask the old question: “Where do we come from?” While this question is usually a philosophical one, in our case it helps to shed some light on our topic. If you do a little reading on the net, you will soon find that pfSense and OPNsense do not like each other much. In fact it’s probably safe to state that they are more or less hostile to each other. OPNsense is a fork of pfSense. Obviously not a friendly fork.

Some pfSense enthusiasts have been spreading information about OPNsense which suggests that new team has no idea what they are doing. They are said to frequently break important things and that the whole project is actually quite laughable. Or to put it short: You really should not waste your time with it and stick to the original. Having liked pfSense for years, I would have believed that, even though you should listen to the other side, too, before doing so. But listening to both sides takes time and effort – both of which were rather limited when I briefly looked into the whole clamor in mid 2015.

Eventually it was the plain hatred of one person who appeared to really have no life, that made me look at what the other project would say. This strange guy popped up in every single pfSense vs. OPNsense discussion and threw so much dirt at OPNsense that I could not help but pity that person. The fact that pfSense (despite obviously being completely dependent on pf, a technology that came from OpenBSD) has a rather bad name with a lot of OpenBSD people for behaving poorly in the past, didn’t help to regain my faith in it, either.

According to OPNsense, they were not happy with the code-quality of pfSense. They didn’t like the fact that the whole Web GUI ran as root (ouch!) and wanted to do privilege separation (which is actively work in progress as I was told). Also there were licensing issues when Netgate acquired pfSense and a bunch of other things. Deciso, a company based in the Netherlands, had been a sponsor of pfSense for years but felt that the whole project was going in the wrong direction after Netgate took a couple of actions. So they decided to fund a fork instead.

Who is right? There’s probably some truth to both versions. OPNsense has followed a rapid development style, bringing in lots of new features and even making some rather drastic changes. It’s true that especially in the beginning there were some problems due to that. But it’s also true that they were quick to react to those. One thing that is not true (or at least not the whole truth, if you will) is that pfSense is the original and OPNsense is a cheap rip-off! What’s the whole truth then?

Once upon a time… in 2003 there was a new firewall OS called m0n0wall. Manuel Kasper had built it on a stripped down version of FreeBSD. There had been small firewalls before, but Kasper’s innovation was to put a Web GUI on top of it so that the firewall’s settings could be controlled from the browser! It did not take long and m0n0wall took the world by storm. However Kasper’s project focused on embedded hardware. So only a while later a fork was created which geared towards more powerful hardware. The fork’s name? You’ve guessed it: pfSense. In 2015 Manuel Kasper officially ended the m0n0wall project (because recent versions of FreeBSD had been grown too big to be easily usable for what he did with it in the past). And guess what he did: He gave his official blessing and recommends to migrate to and support OPNsense!


Knowing some background is nice, but what do both products feel like? The first major difference between the two is what they look like. This is pfSense’s main dashboard:

The pfSense dashboard

Compare it to OPNsense’s version of the dashboard:

The OPNsense dashboard

As you can see, OPNsense did a lot to provide the user a much more modern GUI. Both dashboards are customizable but it’s hard to argue that OPNsense’s is not superior. But to be fair: pfSense is working on a GUI overhaul as well.

Let’s compare a couple of the menus. This is pfSense’s “System” menu:

pfSense: “System” menu

And here’s the one from OPNsense:

OPNsense: “System” menu

While pfSense uses pull-down menus at the top, OPNsense has a navigation bar to the left. As you can see, both have not that much in common. This is because OPNsense did not only redesign the GUI but also re-arranged which options go where. I find the new arrangement more logical (e.g. with pfSense logout is in “System” but halt is in “Diagnostics”). But that’s definitely a matter of taste.

Here’s what the “Services” menu from pfSense looks like:

pfSense: “Services” menu

And here’s the corresponding one from OPNsense:

OPNsense: “Services” menu

This time there seems to be quite a bit more consensus about what counts as a service. But still OPNsense looks more like a cleaned up version.

Another example is the “Diagnostics” menu in pfSense:

pfSense: “Diagnostics” menu

There’s no direct equivalent with OPNsense. In the “Service” menu above you can see that there is a “Diagnostics” entry. The same goes for “System”, “Interfaces”, “Firewall”, etc.

And now for the heart of the whole thing – here’s pfSense’s (default!) WAN firewall rule settings:

pfSense: WAN firewall rules

Compare that to the same thing from OPNsense:

OPNsense: WAN firewall rules

This is where it shows that both products do have a lot in common: What we can see here is basically the same thing. Again OPNsense simply has the more modern interface.

To end the visual comparison let’s look at the LAN firewall rules from pfSense, too:

pfSense: LAN firewall rules

And here’s the LAN rules from OPNsense:

OPNsense: LAN firewall rules

No surprise here: It’s all very similar just with interface improvements on OPNsense’s side.


So far it’s mostly a matter of taste. But now on to the technical points. This is where OPNsense shines (which is no wonder since it’s developing a lot faster). For example OPNsense is already based on FreeBSD 11.0 whereas pfSense is 10.3-based. However there’s already beta versions for the upcoming pfSense 2.4 which are also based on 11.0 and feature many more improvements.

One major difference between the two is that pfSense heavily customized FreeBSD while OPNsense believes in the opposite and tries to be as close to mainline FreeBSD, just adding packages on the top of it. I like that approach better but again that’s probably a matter of taste as well.

The most important thing for me is that OPNsense entered into a partnership with the HardenedBSD project. This resulted in OPNsense being able to change the crypto framework used! For me this is the one killer feature. Give me the option to rip out OpenSSL and use LibreSSL instead and I’m sold! However that’s not even all, yet.

OPNsense: Selecting alternative firmware

OPNsense got HardenedBSD’s ASLR (Address Space Layout Randomization) implementation and the most recent addition is the introduction of packages compiled with SafeStack. This is what the most current update notice looks like (it didn’t fit completely on the screen):

OPNsense: Example of an update notice

If you ask me, hardening your router (especially if it should happen to be promoted to be your border router eventually) makes a lot of sense.

There would be much more to write here, but if you’re interested in that, you will probably have to read some of the recent change notes yourself.


OPNsense and pfSense are quite similar in their core functionality. When should you choose which one? Have a look at the pros and cons of each one and decide for yourself!


  • + Better known brand (= more material about it on the net)
  • + 2.4 will feature BSDinstall including root-on-ZFS
  • +/- Slower development
  • Missing a lot of innovations / security enhancements compared to OPNsense
  • Has been acting quite unfriendly in the past
  • Rather intransparent future direction


  • + Already based on FreeBSD 11.0
  • + Nice new GUI, sensible rearrangement of items
  • + Pushing for priv. sep., uses current technology (PHP 7, phalcon 3, …)
  • + Partnership with HardenedBSD (optional use of LibreSSL, SafeStack, …!)
  • + Closer to unaltered FreeBSD
  • + Bootstrap script to turn a vanilla FreeBSD installation into OPNsense
  • + Multiple languages supported
  • + Public roadmaps for future versions
  • + Official blessings from m0n0wall founder Manuel Kasper
  • +/- Fast-paced development, rapid update policy
  • (Currently) less well-known than its competitor
  • (Currently) installer problems with some USB3 devices
  • Not currently able to install root-on-ZFS

What’s next?

The next article of this series will give an example of an advanced install of OPNsense that lets you use the APU for more than just a router!

Building a BSD home router (pt. 5): Installing OPNsense

Part 1 of this article series was about why you want to build your own router, and how to assemble the APU2 that I chose as the hardware to build this on. Part 2 gave some Unix history and explained what a serial console is. Part 3 demonstrated serial access to the APU and showed how to update its firmware. The previous article detailed installing pfSense.

This post will show how to install OPNsense, a great alternative to pfSense.


OPNsense was forked from pfSense (more on than in the next post) and thus you will find lots of similarities if you have read the post on installing pfSense. The OPNsense team decided to move forward more quickly and did lots of interesting but invasive changes. One strong point for example is that it is already based on FreeBSD 11.0. There is one drawback to this, however: a problem with the XHCI (USB3) driver can lead to the installation media not being able to mount the filesystem and boot up. This makes installing OPNsense a little bit more complicated since the APU2 only has UBS3 ports.

Well, the board does have an internal USB2 controller, too. Therefore I suggest getting a cable that allows connecting USB devices to it. If this is not for you, take a look at the end of the post, I’ve prepared a section “alternative installation methods” there.

First download an image (select amd64 + serial). Then dd it onto an unused memstick and prepare the serial connection (take a look at the previous posts if you need help with dd’ing or attaching the serial console).

Open APU2 box with serial connection and memstick attached to the internal USB2 controller

As you can see, I’ve attached a memstick with OPNsense via USB2 and made a serial connection. That way the installation works just fine.

Step 1: Installation

Hit F10 to go to the boot menu as soon as SeaBIOS offers it.

Boot menu to select which device to boot off of

Since we’ve attached the memstick over USB2, the internal drive would take precedence over it in the default boot order. So in this case I have to select 2 to boot off of the memstick.

The OPNsense boot loader

The OPNsense boot loader looks fine. If you’re installing 17.1 using USB2 you don’t need to do anything here.

Nice feature: Early configuration importer

One notable difference from pfSense is the early configuration importer. If you have a saved configuration XML file, you can put e.g. a UFS2 filesystem on a memstick, create a directory conf on it and copy config.xml there. That makes it available in the importer.

Interface assignment

Then you have the option to assign roles to your interfaces (like WAN and LAN).

Logging into the installer

OPNsense gives you the choice to start the installer or to use a live system. Log in as user installer to perform an installation or as root in the other case. The password for both users is opnsense.

Greeting screen of the installer on the serial console

The OPNsense installer is black and white only when using the console. But that’s fine. The installer greets you with the welcome message.

Console configuration menu

The next screen lets you customize the console. You probably don’t need to do that.

Selecting the installation type

Then you need to select the installation type. You could do advanced partitioning here or setup a softraid (gmirror). We’re going with the simple installation for this post.

Choosing the drive to install on

Now you need to choose where to install to. The mSATA drive is ada0 whereas the memstick is da0.

Selecting the partition scheme to use

OPNsense also lets you choose which partition scheme to use. In case of our router this is not terribly important, especially not with our sample installation that puts everything in one partition. But since stone age is over, you might as well choose GPT anyway.

Progress bar for the installation

While the progress meter was broken with pfSense, this has obviously been fixed for OPNsense. Not that you should reinstall all that often, but still…

Installation done: Reboot!

Once the installation is finished, you of course want to reboot to your new system.

Displaying some information before rebooting

Before rebooting, OPNsense tells you how to access the Web GUI. However the IP address that it uses by default is already taken by my ISP’s modem/router box. We’re going to change that next.

Step 2: Text mode configuration

When the system has started up, you are prompted to log in. This is the default behavior which can be changed to allow unprotected login over the console like with pfSense. But in general I like that bit of extra security.

OPNsense’s text-mode configuration menu

The text-mode configuration menu looks much like that of pfSense.

Configuring the LAN interface

And the interface configuration works right the same.

Setting up DHCP on the LAN interface

As does the DHCP configuration.

Logging out and disconnecting the serial console

Since OPNsense required a login, you can also log out when you’re done. Now disconnect the serial console – we’re done with it.

Step 3: Web GUI configuration

Just like pfSense, OPNsense offers a nice Web GUI to configure all the settings. Fire up your browser on a PC that is in the same subnet (or got its IP address via DHCP from the new router) and enter the router’s LAN IP address in the URL bar.

Self-signed certificate warning

OPNsense uses https to create a secure connection, too. Of course a self-signed certificate is used which is not trusted by my Firefox. Therefore a permanent exception needs to be made.

OPNsense Web GUI login screen

Once you have confirmed the exception, you will see the login screen. Log in as root with the password opnsense.

The configuration wizard

On the first login you will be greeted by the configuration wizard. It will present you about the same choices as pfSense does (without the advertizing of the commercial version, of course).

Configuring general settings

First it’s some general information like hostname and DNS. What OPNsense offers over pfSense is i18n options: Chances are that you can configure the Web GUI to speak your language! That’s pretty nice.

Configuring time-server settings

Time server settings are just like those from pfSense.

Configuring the WAN interface

WAN configuration offers you a lot of options. Take a close look at those. Fortunately you very likely don’t need most of what is there.

Configuring the LAN interface

Same thing for the LAN configuration: You know that from pfSense.

Setting a new password for the Web GUI

Also with the password changing part there’s no surprise here.

All done. Reload the config!

That’s it. Reload the config now and you’re done with the wizard. OPNsense now has a basic configuration and is ready to be used.

Alternative installation methods

OK, you don’t have a cable to connect to the USB2 pins but you want OPNsense? There are several things that you can try. I’ve documented my attempts (including several solutions) on the OPNsense forums in case anybody needs them.

Here are a few things that you can try:

  • Install from SD card (I didn’t try that but it should indeed work)
  • Install 16.7 from USB3 with increase boot_delay and then update
  • Install 17.1 using a USB cdrom, manually enabling the console and importing a pre-made configuration

Should you install 16.7 using a USB3 port, press ESC before the loader countdown runs out. This will drop you to the loader prompt. Then enter the following:


That did the trick and made the system boot up for me. The actual installation is quite similar to what I covered above.

You could also use a USB cdrom to boot the installation – of course use the OPNsense cdrom ISO in this case! However the cdrom image does not have the serial console enabled by default. So escape to the loader prompt, set some variables to enable the serial console and boot:

set boot_multicons=YES
set boot_serial=YES
set comconsole_speed=115200
set console=comconsole,vidconsole

This will work, too. But there’s one little problem with that: The TTYs are configured on their own using a configuration file – and they are not ready for serial connection! Since this is a CD, we cannot really do much about that. What we can do, however, is using the configuration importer. I will upload a basic configuration xml and add it to this post when I next install a clean OPNsense.

What’s next?

The next post will be pfSense vs. OPNsense! It will discuss some of the notable differences and when to use which one.

Building a BSD home router (pt. 4): Installing pfSense

Part 1 of this article series was about why you want to build your own router, and how to assemble the APU2 that I chose as the hardware to build this on. Part 2 gave some Unix history and explained what a serial console is. Part 3 demonstrated serial access to the APU and showed how to update its firmware.

This post is about the serial installation of pfSense, one of two FreeBSD-based router/firewall operating systems that we’re going to explore in this series (the other being OPNsense). As pfSense is the older and more established product, we’re beginning with that one.


We’re just doing the installation here. A closer look at using pfSense or a comparison with OPNsense will be another post. Getting pfSense up and running is really easy, even when you’re using the serial console. The first step is the actual installation. In a second step you need to configure the LAN interface and then you can use the WebGUI to do the final setup.

The first thing to do, however, is getting preparing an installation medium. Head over to pfSense’s Download site. What you want is an install image for amd64. Then select USB Memstick Installer which let’s you choose the console type – obviously get the serial one!

Then get a USB stick that you can spare and dd the image on it. Once you have that ready, plug it into the APU. Next attach the serial cable to your APU and to another computer. Then connect to the console (how to do that was described in the previous post). Now power on the APU.

Step 1: Installation

Even if there’s already an OS installed on your mSATA drive, the memstick should take precedence when it comes to boot order. So you can probably just wait until the installer comes up.

pfSense’s loader menu: screwed up over the serial console…

Don’t be scared when you see garbage displayed on the screen. This is just the bootloader that’s screwed up badly when used over a serial connection (they’ve already fixed that in the beta version for the upcoming pfSense 2.4). Either just wait 10 seconds for it to boot automatically or press enter to boot right now (if you need any other options, you might want to get an ISO for pfSense, too, and test it in a VM or get a VGA image, put that on a stick and try it out on hardware that provides a local console over a screen and keyboard).

…but once the kernel loads, text is fine

As you can see, it’s only the loader. As soon as the kernel takes over, the text is displayed correctly. That means you can actually read the messages in case anything goes wrong here. If you don’t do anything, the installer will eventually come up automatically.

First screen of the installer

In the first screen of the installer you can configure the console. Most likely the defaults will be fine, though.

Selecting the installation method

Then you need to choose the installation method. We will do a quick installation but you could also do a custom installation or setup gmirror (mirrored software RAID).

The usual “this will erase your data” warning

Since installing pfSense means destroying any data that might currently be on the drive, the installer warns you that it will erase it.

Installation progress bar

If you confirmed the warning, the actual installation starts (but the progress meter is kind of useless as it seems… It remained at 5% for a while and then jumped to 100% for me).

Kernel selection

The next thing to do is to select the right kernel. Since our APU2 is a headless device, make sure that you select the embedded kernel! Otherwise you won’t be able to use the serial console with it.

Another progress bar

After the kernel is installed, the installer runs a script to do some final tasks.

Reboot message

When all is done, it’s time to reboot the system.

pfSense rebooting after installation

Just before it reboots, pfSense prints some important information on the screen, telling how to log into the WebGUI. Remove the memstick now or the APU will boot off of it once more an you’ll just see the installer again.

Step 2: Text mode configuration

The OS has been successfully installed, but leave your serial console attached for now.

pfSense’s text mode management menu

Once the system has booted, you will see the management menu. It offers a lot of tools including going to a shell (option 8) and doing everything you like. We want to configure the IP address for our LAN interface (option 2):

Configuring the LAN interface

I’m assigning since my modem/router (yes, I’m not replacing it just yet and will operate the new router between that box and my actual network for now) has already taken It’s not like I need a full /24 subnet for my network, but I go with that subnet mask for now.

Configuring DHCP for the LAN interface

Since I intend to use DHCP for my network, I enable a DHCP server for the LAN interface. The range of DHCP addresses that I use here is just an example for this test installation. I will cut it down to about 10 when I do my final setup. The reserved addresses before the DHCP range serve a purpose, though – more on that in a separate future post.

Back at the menu

As soon as everything is ready, you can now end the serial connection and remove the cable. We have a valid IP address on the LAN interface now after all.

Step 3: WebGUI configuration

So now we can access the WebGUI simply by entering the IP address in the URL bar of any browser. Of course the computer that runs the browser have an IP address that is on the same subnet. So you might want to change your address if that is not the case – or fire up the dhclient, it should get an address in the range that you specified (or simply reboot if your computer is configured for DHCP).

Self-signed certification warning

It’s a good thing that pfSense uses TLS so you can access the router securely via https. However the certificate it uses is self-signed and thus unknown to your browser which will display a warning. That doesn’t mean that it’s useless. In our case it’s just necessary to create an exception to accept that cert permanently.

Logging into pfSense’s WebGUI

You’ll then see the login screen. Use the username admin and the password pfsense to log in.

Running the configuration wizard

Once you’re logged in, pfSense suggests that you run the configuration wizard – and that makes sense.

A little advertising for pfSense Gold

The first screen of the wizard is an advertisement for the commercial version of pfSense called pfSense gold. If you are a company looking for more than the free “Community Edition” of pfSense will give you, have a look at this service. Maybe it’s for you.

General information configuration

First you configure some general settings like the hostname, domain, etc.

Time Server configuration

Next is the configuration of the time zone and NTP daemon.

WAN configuration

Then the WAN interface needs to be configured. There are a lot of settings there and very likely you don’t need all of them.

LAN configuration

After that comes the LAN interface. Here you can only configure the IP address and subnet mask (which we already did in text mode).

Changing the password for the WebGUI

Finally we’re prompted to change the password which is a good idea of course. Even if the WebGUI is only accessible from the LAN interface by default, it’s a matter of principle.

Configuration done: Reload!

That’s it, the wizard is finished. Time to reload the configuration.

All done, pfSense is ready

We’re done here, pfSense is installed and the basic configuration has been applied. There’s another little advertising here which is legit for a free product, I guess. We’re going to take a look at the main WebGUI and its many, many options in another post.

What’s next?

The next blog post will detail the installation of OPNsense, another excellent option for your router.

Building a BSD home router (pt. 3): Serial access and flashing the firmware

Part 1 of this article series was about why you want to build your own router, and how to assemble the APU2 that I chose as the hardware to build this on. Part 2 gave some Unix history and explained what a serial console is.

In this post we will prepare a USB memstick to update the BIOS and connect to the APU2 using the serial console. Then we’ll flash the latest firmware on there.

Cables and serial connections

In the ol’ days you would simply connect the COM port on one machine to the COM port on the other. Today a lot of newer laptops don’t even have a serial port (if yours still has one of those funny devices that you’d access through /dev/fd0, chances are pretty high though, that it also has a COM port!). Fortunately USB to serial adapter cables exist, solving that problem.

The APU2 has a male DB9 (9 pins) serial port. RS-232 is the common standard for serial communication. According to it, some pins are used to transfer information while others are used to receive information. Now if you connect two machines with a straight serial cable, both will talk on the same pins and listen on the same pins. So both will send data over pins that nobody listens on and never receive anything on the other pins. This is not really useful. To make the connection work, you need a crossed-over cable (a so-called null modem cable or adapter). This means that the receiving pins on one end are paired with transmitting pins on the other end and vice versa.

I thought that I would never need a nullmodem cable at home. I still don’t think that I might ever need a straight serial cable. And I could in fact take one home from work and return it the other day. However I could already see what would have happened in that case: When I get some time to tinker with my APU it will be on the weekend and I won’t have the cable in reach when I need it. So I got my own. And while I was at it, I decided to not only get a USB to RS-232 DB9 serial adapter cable (look for the PL2303 chipset translating USB to serial: It’s well supported across a wide range of operating systems including FreeBSD). I also bought a null modem adapter and a gender changer. So now I’m completely flexible with my gear. However you probably just want to get a null modem cable USB/female DB9 (or ask somebody who has one if you could borrow it).

Another thing that you have to know is the baud rate (modulation rate) for your connection. A higher baud rate means a faster connection. As long as both connected machines agree on the baud rate, everything is fine. If they disagree, this can lead to displaying garbage instead of the actual console or in seemingly nothing happening at all.

To flash or not to flash

At the time of this writing, PC Engines have released 5 updates for the APU2’s firmware and if you like the improvements, it makes sense to put the newest version on there. They recommend booting TinyCore Linux and then using flashrom to flash the BIOS.

Flashrom is available for FreeBSD, too. I would imagine that it works as well. However I have no experience with that and flashing stuff always bears the risk of bricking your device (for which case PC Engines offers a small rescue device). If I had thought about this right at the beginning, I would probably have tried it out. But my APU2 is already updated and since this post is public and not just for me… Well, let’s just do this by the book and use Linux for that.

If you’re a little anxious and don’t feel well about flashing at all, leave it be; in general the old BIOS will do, too. Flashing according to a guide does not have a high risk of bricking your device but even a small risk is a risk. Disclaimer: You decide. And it’s all your responsibility, of course.

Alright, we need to prepare a USB stick with TinyCore and the ROM on it. PC Engines even offer a howto guide showing how to create this using FreeBSD. That guide works, but it clearly shows that those guys know Linux a lot better than (modern) FreeBSD. For that reason I’m going to modify it slightly here and use today’s tools.

Preparing the BIOS updater memstick

First we’re going to download the TinyCore tarball and the zipped ROM (you might want to take a look if a newer version than shown here is out) and install the syslinux package (containing the that Linux bootloader):

% su -
# mkdir -p /tmp/apu2 && cd /tmp/apu2
# fetch
# pkg install syslinux

Now attach your USB memstick to the system and after a couple of seconds look at the dmesg if you don’t know what device it will be attached to:

# dmesg | tail
da0 at umass-sim0 bus 0 scbus2 target 0 lun 0
da0: < USB DISK 2.0 PMAP> Removable Direct Access SPC-4 SCSI device

So for me it’s da0 in this case and I need to supplement the “X” that I use in the following commands with “0”. You should use whatever number you figured out. PC Engine’s howto suggests zeroing out the stick and if it isn’t a new and unused one that makes sense. In my case this leads to an error:

# dd if=/dev/zero of=/dev/daX bs=1m
dd: /dev/da0: Operation not supported

# mount
/dev/da0s1 on /media/disk (ext2fs, local, nosuid)

Looks like this is because GhostBSD automatically mounted da0s1 when it found an EXT2 filesystem from a previous task in there. So let’s unmount that and try again:

# umount /media/disk
# dd if=/dev/zero of=/dev/daX bs=1m

Depending on the speed and size of your memstick (and the generation of your USB port) this can take quite some time to finish. And since dd normally is working quietly, you might want to know how far it came so far. Fortunately FreeBSD implements the SIGINFO signal. Just press CTRL+T while it is running and it print some status information like this:

load: 1.52  cmd: dd 92084 [physwr] 379.15r 0.00u 0.56s 0% 3188k
2512+0 records in
2512+0 records out
2634022912 bytes transferred in 379.208179 secs (6946113 bytes/sec)

When it’s done, we can create an MBR partitioning scheme on the now empty stick as well as an MBR partition and write the bootcode on the stick so that we can boot off of it at all:

# gpart create -s mbr daX
da0 created
# gpart add -t fat32 daX
da0s1 added
# gpart bootcode -b /boot/boot0 daX
bootcode written to da0

The next thing to do is putting a FAT32 filesystem on the partition and force install the Syslinux bootloader there that will be chainloaded by the bootcode that we wrote into the MBR.

# newfs_msdos /dev/daXs1
newfs_msdos: trim 40 sectors to adjust to a multiple of 63
# syslinux -if /dev/daXs1

Now we need to mount the new filesystem and put the OS on there:

# mount -t msdosfs /dev/daXs1 /mnt
# tar xjf apu_tinycore.tar.bz2 --no-same-owner -C /mnt

We’re writing to a FAT filesystem – and as the primary filesystem from a once popular single user OS it does simply not support concepts like file ownership. That’s why we need “–no-same-owner” here (otherwise we’d see harmless but unnecessary warnings). As the next step we’ll add the ROM image and check its integrity – we don’t want to flash garbage on our APU and brick it, do we?

# unzip -d /mnt
# grep -c `md5 -q /mnt/apu2_v4.0.7.rom` /mnt/apu2_v4.0.7.rom.md5

Make sure that the last command outputs 1 (in which case the calculated md5 hash matches)! If it does not, delete the zip file, download it again and extract it over the corrupt ROM file. Finally sync the filesystem, unmount it and remove the USB stick:

# sync
# umount /mnt

The memstick is ready. If you want to test it, boot some other PC or laptop from it. If you can read the line “Booting the kernel” and then nothing seems to happen anymore, it means that it’s working. TinyCore is configured to use the serial console, that’s why you don’t see anything on your screen and your keyboard doesn’t do anything after that. Just turn your computer off and plug the USB stick out.

Attaching the serial console and flashing the BIOS

Alright, back to the APU2 (finally). Put the memstick into one of the USB ports and attach your serial cable to the COM port. Now connect the other end of the null modem cable with another computer running FreeBSD (or Linux or whatever – this should all work, you’ll just have to figure out how to connect to a serial console on your OS).

Open a terminal, become root and attach the serial console (cuaU0 is the USB to serial adapter, 115,200 the baud rate):

% su -
# cu -l /dev/cuaU0 -s 115200

Now connect the APU2 to power and see what happens! If you don’t do anything, the BIOS should load TinyCore from the memstick after a couple of seconds:

Connected to the serial console – and booting Linux

If nothing happens, you might have the wrong cable (is it really crossed-over?). Or maybe you’ve mistyped the baud rate? The “Connected” by the way only means that your host system has attached to the serial cable. You’ll get that message even when the other end is not connected to anything or the machine at the other end is turned off.

Once Linux was loaded, use flashrom to update the APU’s firmware like PC Engines show in their howto. Then reboot.

Preparing to flash the BIOS

The APU2’s BIOS supports the serial console. That means that even before the machine has booted an operating system with serial console capability, you can access and configure the BIOS of the headless machine:

Serial access to the BIOS settings

Another nice thing that comes with the APU is the Memtest program. If you want to know whether your new hardware is actually good or might probably have bad ram, put it to the test for a couple of hours or over night:

The firmware comes with Memtest

If you’re using cu as in my example, you can close the serial connection using the character sequence ~. (tilde and dot).

What’s next?

You now know how to access your box using the serial console. Next time we’ll make use of that skill again to put pfSense as the first of two options on the APU. The other option is OPNsense which will be covered in a later post. Both are FreeBSD-based router/firewall operating systems.

Building a BSD home router (pt. 2): The serial console (excursion)

The previous post touched on the topic of why you might want to build your own router, discussed some hardware considerations and showed how to assemble an APU2 bundle. Now that the box is ready for use, we have to do something with it – but there’s a small problem.

People who want to play with inexpensive hardware love their Raspberry Pis. Those make a great little toy: Just attach a keyboard and a monitor, insert an SD card and you’re done. Power on the device and you can start tinkering. With the APU2 it’s not that much harder actually, but it works quite differently. Why? Take a closer look at its backc and you’ll see that the outer left port is not VGA as you might have thought when you saw it from a different angle. It’s a COM port. There are two USB ports which would allow to attach a keyboard (the board doesn’t have a keyboard controller, though, which means that wouldn’t be of much use). But how to attach a screen to it? The RPi offers HDMI output, the APU2 does not. No VGA, no DVI, no HDMI and no DisplayPort either!

So how to access this thing? Typing blindly? Of course not. This post is about the serial console and since they call the COM port a serial port, too, it’s not too hard to figure out what we’re going to use. If you were born after 1980 (like me), chances are that you’ve never used a serial console before. Or rather you might not know exactly what it actually is. Heck, the younger ones might not even have heard about it! So if you’re either old enough or for whatever reason have experience with the serial console you could skip a lot of the following text. If you know what the real TTYs were, what a terminal and a system console is, this whole post probably isn’t going to teach you anything new. But since there are more and more younger people around for whom this is entirely new territory, I’m going to explain not only what we have to do in case of the APU2 but also a bit about how this all came to be.

Once upon a time… in 2017

Huh, 2017? That’s this year! Exactly. But then this headline doesn’t make sense, does it? Trust me, it does. Consider this:

% uname -a
FreeBSD pc0.local 11.1-PRERELEASE FreeBSD 11.1-PRERELEASE #0 r319447: Fri Jun  2 03:47:52 CEST 2017     root@pc0.local:/usr/obj/usr/src/sys/GENERIC  amd64

% ls /dev/tty*
/dev/ttyu2      /dev/ttyv0      /dev/ttyv3      /dev/ttyv6      /dev/ttyv9
/dev/ttyu2.init /dev/ttyv1      /dev/ttyv4      /dev/ttyv7      /dev/ttyva
/dev/ttyu2.lock /dev/ttyv2      /dev/ttyv5      /dev/ttyv8      /dev/ttyvb

Here’s an up-to-date FreeBSD system and it has quite some device nodes that bear “tty” in their name. The abbreviation TTY means Teletypewriter (or “teletype” for short) and with that we’re speaking of a technology that has its roots far back in the first half of the 19th (!) century. As I stated in my first post about FreeBSD 4.11 and legacy systems: Once some technology is established and in use, it’s extremely hard to get rid of it again. And while it’s hard to believe that concepts from almost two hundred years ago still influence today’s Unix machines, there’s some truth to it.

So what is a teletypewriter? Think a classical typewriter first. One of those mechanical machines that allow to type text instead of writing it: You press a key and that makes a small metal arm swing and its end hammer against the paper. That end bears an embossed letter. Between this head with the letter (or rather: character) and the paper there’s an ink ribbon and thus your key press results in a readable imprint on the paper.

Man has long striven to be able to transfer information across a long distance. Morse code made it possible to encode text and then send a message e.g. via optical means like blinking light. The telegraph (from Greek tele + graphein = far + write) achieves the same goal, however by sending signals over a cable.

A simple teletypewriter combines both: Imagine a typewriter with all the usual keys but no paper that is connected to a remote device which has the paper feed, ink ribbon and so on but no keys. This second device is called a teleprinter. The big limitation to this is that the information flow is one-way only.

Where computers fit into this

Sounds familiar? Sure thing: The keyboard that you use essentially does exactly that! Now consider a more advanced approach where two teletypewriters that both have input and output capabilities are connected to each other. In this bi-directional use you can type something that the person on the other end can read as well as write an answer which then you can read (and write again).

It’s obvious how this supports distributing information across large distances. But what does all this have to do with the COM port? Please bear with me. The first type of computers were big and extremely expensive mainframe machines. It makes sense that not everybody would be allowed physical access to them. Thus you wouldn’t sit down before them and attach a screen and keyboard! Instead you’d use a teletype in another room to interact with it. You input any commands and get a response in form of the printed output. More sophisticated teletypes allowed saving input or output on punch cards or read them in again and send the data.

Whatever combination of gear that allows sending and receiving data to/from a computer is called a terminal. Bi-directional typewriters are sometimes referenced as “hard-copy terminals” (because they printed the output on paper). While it’s hard to believe, this procedure has been state of the art at some point in time. Obviously this is not ideal, wasting a lot of paper. Fortunately there’s a solution for this: Enter the electronic terminal! Those replaced the printing on paper with displaying on a screen.

Now this is much more of what we’re familiar with, just in a special machine that is connected to the actual computer in a different place. The great thing about those terminals was that they allowed for adding more and more exciting new features. The bad thing was that they added more and more exciting new features. Usually there was no gain in having those features on the client side only. The server side had to know about it, too. Before long there were a lot of terminals out there, all with nice new features but all incompatible with each other…

This sucked. If you had this expensive new terminal that was unknown to your computer however, you couldn’t use its additional features. Good thing that we don’t use terminals today! That makes this an issue of the past. Right, that is a good thing. But no, it’s not a historic issue. Yes, we’ve replaced terminals quite a while ago. But what did we replace them with? Today we use terminal emulators. Your *nix operating system comes with programs that (by default) use the keyboard for input and the screen for output and thus simulate… a terminal.

Terminal emulators

Those terminal emulators can emulate various types of physical terminals and have additional features of their own. So the problems are back (or rather: They were never really gone). How do programs know what features are available? That used to be really tricky (and often lead to discarding features that were not commonly available on all popular terminals…). Let’s take a look at my shell’s environment:

% env

My shell is running inside the terminal multiplexer tmux which set the TERM variable to “screen”. GNU screen was a somewhat similar program and since tmux is compatible with it, it announces that terminal-wise it works like screen. What does this look like if I quit tmux?

% env

Here my terminal emulator claims to support the feature set that xterm is known for. I’m not using xterm, but my terminal emulator (terminator) is compatible with it. You probably know htop. I’ve just installed it to make two screenshots of it. This is what it looks like if I run it in my terminal emulator:

htop running with $TERM=xterm

Now let’s claim that we’re using another terminal type, shall we? I’m going to pretend that my terminal is an old, crufty vt100:

% setenv TERM vt100
% htop

Afterwards the application looks like this:

htop running with $TERM=vt100

The difference is pretty obvious – the vt100 did not support color output (among other things). Let’s be even more radical and simply not announce our terminal type to any applications:

% unsetenv TERM
% tmux
open terminal failed: terminal does not support clear

Tmux won’t even run (and htop would screw it all up badly since it obviously doesn’t handle that case properly)! If the terminal type (and thus its capabilities) are unknown, programs need to fallback to the absolute baseline of functionality that each and every terminal is known to support. And that’s disturbingly little… In this case tmux would be unusable because the terminal might not even be able to clear the screen! Yes, not even the seemingly most basic things were implemented in all terminals.

Imagine you’re the author of a simple *nix program. You were supposed to know what terminals existed and what capabilities they had – and to adopt your program to whatever terminal it ran on! Fortunately Bill Joy wrote termcap, a library that helped dealing with various terminals. This extremely important functionality (which originated in BSD Unix, BTW!) evolved over time. Today we have ncurses which does just about everything to make life easy when programming console applications.

(Serial) console

Which leads to the question: What is the console? The system console (or just console) is a device used to imput and output text. A PC’s BIOS offers a console, the boot loader does and the OS kernel does, too: It uses it to print out messages and to take commands. It’s present even when you don’t see it. A system might for example work in so-called headless mode which means that there is nothing attached to the console. Plug in a keyboard and connect a monitor and you can make use of the console.

Unix-like operating systems often configure multiple virtual consoles (also called: virtual terminals). We saw those when we listed the tty* nodes in /dev at the beginning of this blog post. Together those form your computer’s system console to which usually keyboard and screen are attached.

But you need not attach those two devices to the console. You can also attach a serial device to it which can then be used to transfer commands from and messages to another computer that is connected to the other end of the serial cable. Then you can use that other computer’s keyboard to type commands and its screen to read the output. A serial console is simply a console that can make use of a serial connection to redirect the console. If the BIOS of your machine supports it, you can use the serial console to control everything on that machine using another one over a serial connection.

If you’re completely new to it, this probably was quite a bit of information. However we’ve only talked about the very basics here! If you like to know more about how TTYs, pseudo TTYs, processes, jobs and even signals work, I recommend Linus Ã…kesson’s article The TTY demystified which is an excellent read. He explains much more in-depth how Linux does all this.

What’s next?

In the next post we’ll prepare a USB memstick and attach a serial console to the APU2 to update the firmware.