A SPARC in the night – SunFire v100 exploration

While we see a total dominance of x86_64 CPUs today, there are at least some alternatives like ARM and in the long run hopefully RISC-V. But there are other interesting architectures as well – one of them is SPARC (the Scalable Processor ARChitecture).

This article is purely historic, I’m not reviewing new hardware here. It’s more of a “20 years ago” thing (the v100 is almost that old) written for people interested in the old Sun platform. The intended audience is persons who are new to the Sun world, who are either to young like me (while I had a strong interest in computers back in the day, I hadn’t even finished school, yet, and heck… I was still using Windows!) or never had the chance to work with that kind of hardware in their professional career. Readers who know machines like that quite well and don’t feel like reading this article for nostalgic reasons might just want to skip it.

The SPARC platform

SPARC is a Reduced Instruction Set Computing (RISC) Instruction Set Architecture (ISA) developed by Sun Microsystems and Fujitsu in 1986. Up to the Sun-3 series of computers, Sun had used the m68k processors but with Sun-4 started to use 32-bit SPARC processors instead. The first implementation is known as SPARCv7. In 1992 Sun introduced machines with v8, also known as SuperSPARC and in 1995 the first processors of SPARCv9 became available. Version 9, known as UltraSPARC, is a 64-bit architecture that is still in use today.

SunFire v100: Top and front view

SPARC is a fully open ISA, taken care of by SPARC International. Architecture licenses are available for free (only an administration fee of 99$ has to be payed) and thus any interested corporation could start designing, manufacturing and marketing components conforming to the SPARC Architecture. And Sun did really mean it with OpenSPARC: They released the Verilog code for their T1 and T2 processors under the GPLv2, making them the first ever 64 bit processors that were open-sourced. And not enough with that – they also released a lot of tools along with it like a verification suite, a simulator, hypervisor code and such!

After Sun was acquired by Oracle in 2010, the future of the platform became unclear. Initially, Oracle continued development of SPARC processors, but in 2017 completely terminated any further efforts and laid off employees from the SPARC team.

Fujitsu has made official statements that they are continuing to develop the SPARC-based servers and even about a “100 percent commitment”. In the beginning of this year, they even wrote about a Resurgence of SPARC/Solaris on the company’s blog and since they are the last one to provide SPARC servers (which are still highly valued by some customers), chances are that they will continue improving SPARC. According to their roadmap, even a new generation is due for 2020.

So while SPARC is not getting a lot of attention these days, it’s not a dead platform either. But will it survive in the long run? Time will tell.

SunFire v100

I’m working for company that offers various hosting services. We run our own data center where we also provide colocation for customers who want that. Years ago a customer ran a root server with an (now) old SunFire v100 machine. I don’t remember when it was decommissioned and removed from the rack, but that must have been quite a while ago.

SunFire v100: Back view

That customer was meant to come over to collect the old hardware and so we put the machine in the storage room. For whatever reason, he never came to get it. Since it had been sitting there for years now, I decided to mail the customer and asked if he still wanted the machine. He didn’t and would in fact prefer to have us to dispose of it. So I asked if he’d be ok with us shreddeing the hard drives and me taking the actual machine home. He didn’t have any objections and thus I got another interesting machine to play with.

The SunFire v100 is a 1U server that was introduced in 2001 and went EOL in 2006. According to the official documentation, the machine came with 64 bit Solaris 8 pre-installed. It was available with an UltraSPARC IIe or IIi processor and had a 40 GB, 7200 RPM IDE HDD built-in. My v100 has 1GB of RAM and a 550 MHz UltraSPARC IIe. I also put a 60 GB IBM HDD into it.

It has a single PDU, two ethernet ports as well as two USB ports. It also features two serial ports – and these are a little special. Not only are they RJ-45, but they have two different uses cases. One is for the LOM (we’ll come to that a little later), the other one is a regular serial port that can be used e.g. to upload data uninterrupted (i.e. not going to be processed by the LOM). The serial connection uses 9600 baud, no parity, one stop bit and full duplex mode.

RJ-45 to DB9 cable and DB9 to USB cable

The other interesting thing is the system configuration card. It stores host ID and MAC address of the server as well as NVRAM settings. What is NVRAM? It’s an acronym for Non-Volatile Random-Access Memory, a means for storing information that must not be lost when the power goes off like regular RAM does. If you’re thinking “CMOS” in PC terms, you’re right – except it seems that Sun used a proper means of NVRAM and not an in fact volatile source made “non-volatile” by keeping the data alive with the help of a battery. The data is stored on a dedicated chip, or in this case on a card. The advantage of the latter is that it can be easily transferred to another system, taking all the important configuration with it! Pretty neat.

Inside the v100

When I opened up the box, I was actually astonished by how much space there was inside. I know some old 1U x86 servers from around that time (or probably a little later) that really are a pain to work with. Fitting two drives into them? It’s sure possible, but certainly not fun at all. At least I hated doing anything with them. And those at least used SATA drives – I haven’t seen any IDE machines in our data center, not even with the oldest replacement stuff (it was all thrown out way before I got my job). But this old Sun machine? I must say that I immediately liked it.

SunFire v100: Inside view

Taking out the HDD and replacing it with another drive was a real joy compared to what I had feared that I’d be in for. The drive bays are fixed using a metal clamp that snaps into a small plastic part (the lavender ones in the picture). I’ve removed the empty bay and leaned it against the case so that it’s easier to see what they look like. It belongs where the ribbon cable lies – rotated 90 degrees of course.

Old x86 server for comparison – getting two drives in there is very unpleasant to do…

All the other parts are easily accessible as well: The PDU in the upper left corner of the picture, the CDROM drive in the lower right, as well as the RAM modules in the lower left one. It’s all nicely laid out and well assembled. Hats off to Sun, they really knew what they were doing!

Lights out!

I briefly mentioned the LOM before. It’s short for Lights-Out Management. You might want to think IPMI here. While this LOM is specific to Sun, its basic idea is the same as the wide-spread x86 management system: It allows you do things to the machine even when it’s powered off. You can turn it on for example. Or change values stored in the NVRAM.

LOM starting up

How do we access it? Well, the machine has a RJ-45 socket for serial connections appropriately labeled “LOM”. The server came with two cables to use with it, one RJ-45 to DB26 (“parallel port”) used with e.g. a Sun Workstation, and a RJ-45 to DB9 (“serial port” a.k.a. “COM port”). Then you can use any of the various tools usually used for serial connections like cu, tip or even screen.

Just plug your cable into say your laptop and the other end into the A/LOM port, then you can then access the serial console. If you plug in the power cable of the SunFire machine now, you will see the LOM starting up. Notice that the actual server is still off. It’s in standby mode now but the LOM is independent of that.

LOM help text

By default, the LOM port operates in mixed mode, allowing to access both the LOM and the serial console. These two things can be separated if desired; then the A port is dedicated to the LOM only and the console can be accessed via the B port.

In case you have no idea how to work with the LOM, there’s a help command available to at least give you an idea what commands are supported. Most of these commands have names that make it pretty easy to guess what they do. Let’s try out some!

LOM monitoring overview (powered off)

Viewing the environment gives some important information about the system. Here it reveals that ALARM 3 is set. Alarm 1, 2 and 3 are software flags that don’t do anything by themselves. They can be set and used by software installed on the Solaris operating system that came with the machine.

I really have no idea why the alarm is set. It was that way when I got the server. Even though it’s harmless, let’s just clear it.

Disabling alarm, showing users and booting to the ok prompt

The LOM is pretty advanced in even supporting users and privileges. Up to four LOM users can be created, each with an individual password. There are four privileges that these can have: A for general LOM administration like setting variables, U for managing LOM users, C to allow console access as well as R for power-related commands (e.g. resetting the machine). When no users are configured, the LOM prompt is not protected and has full privileges.

OpenBoot prompt

It is also possible to set the boot mode in the LOM. By doing this, the boot process can e.g. be interrupted at the OpenBoot prompt which (for obvious reasons) is also called the ok prompt. In case you wonder why the command is “boot forth” – this is because of the programming language Forth which the loader is written in (and can be programmed in).

ok prompt help

In the ok prompt you can also get help if you are lost. As you can see, it is also somewhat complex and you can get more help on the respective areas that interest you.

Resetting defaults and probing devices

OpenBoot has various variables to control the boot sequence. Since I got a used machine, it’s probably a good idea to reset everything to the defaults.

From the ok prompt it’s also possible to probe for devices built into the server. In this case, an HDD and a CDROM drive were found which is correct.

Setting NVRAM variables, escaping to LOM, returning to the ok prompt and resetting the machine

The ok prompt allows for setting variables, too, of course. Here I create an alias for the CDROM drive to get rid of working with the long and complex device path. Don’t ask me about the details of the latter however. I found this alias on the net and it worked. I don’t know enough about Solaris’ device naming to explain it.

Next I set the boot order to CDROM first and then HDD. Just to show it off here, I switch back to the LOM – using #. (hash sign and dot character). That is the default LOM escape sequence, however it can be reconfigured if desired. In the LOM I use the date command to display how long the LOM has been running and then switch back to the ok prompt using break.

LOM monitoring overview while the machine is running

Finally I reset the machine, so that the normal startup process is initiated and an attempt at booting from the CDROM is being made. I threw in a FreeBSD CD and escaped to the FreeBSD bootloader (which was also written in Forth until it was replaced with a LUA-based one recently).

Showing the monitoring overview while the machine is actually running is much more interesting of course. Here we can see that all the devices still work fine which is great.

LOM log and date, returning to console and powering off

Finally I wanted to show the LOM log and returning to the console. The latter shows the OK prompt now. Mind the case here! It’s OK and not ok. Why? Because this is not the OpenBoot prompt from the SunFire but the prompt from the FreeBSD loader which is the second-stage loader in my case!

That’s it for the exploring this old machine’s capabilities and special features. I just go back to the LOM again and power down the server.

Conclusion

The SunFire v100 is a very old machine now and probably not that useful anymore (can you say: IDE drive?). Still it was an interesting adventure for me to figure out what the old Sun platform would have been like.

While I’m not entirely sure if this is useful knowledge (SPARC servers in the wild are more exotic then ever – and who knows what the platform has evolved into in almost 20 years!), I enjoy digging into Unix history. And Sun’s SPARC servers are most definitely an important mosaic in the big picture!

What’s next?

Reviewing this old box without installing something on there would feel very incomplete. For that reason I plan to do another article about installing a BSD and something Solaris-like on it.

Thoughts on being a “power user” (1/3)

Every so often I come across the term “power user”. It’s used to describe people who are more knowledgeable and capable than the casual or regular users are. Now it’s pretty obvious that this definition leaves a lot of room in considering somebody a power user or not. People seem to generally agree you don’t become one by just memorizing a couple of hotkeys. However it doesn’t require anyone to reach perfection if you ask me. Anybody being clearly way above average qualifies.

Oh, and of course you can be a power user in one field while remaining a complete novice in other areas. I’m somewhat familiar with *nix operating systems and have invested quite a bit of time in becoming a better user. If I’m to do graphics manipulation however, it doesn’t make so much of a difference for me whether I use the GIMP or Tux Paint as that simply isn’t my field of expertise at all…

In this three-post article I’ll write about some of the things that might matter to anybody who would like to make more of his or her PC-related skills. A lot of that depends on what you are actually doing, so not everything that I mention might apply to the same degree for you. I’m a sysadmin by profession, but I know a bit of a programmer’s perspective, too. Still I think the interested hobbyist or people who work with computers all day will find some tips on how to increase efficiency in their workflows. Especially the points in this first post are pretty general and should apply to just about anybody since it’s about hardware.

1) Touch typing – and considering your layout

An obvious activity to improve your skills in using a computer is to learn touch typing. I recommend to learn it as early as possible: It’ll help you through most of your life after you start doing it. Also it’s not as hard as it might seem. Decide that you want to do it and practice for a few weeks. That’s all. It’s completely sufficient to learn to type without looking at the keyboard. Even if you stop practicing there, your speed will improve as you keep typing. If you enjoy it, by all means do practice speed at least a little! There’s a lot to be gained here. But basic touch typing is enough to be counted as a valuable skill. You’ll get faster just by doing it anyway.

There is free training software out there (I recommend klavaro). But before you run off to learn it, do a little research on the net first! Learning QWERTY (or whatever the default layout in your country is) certainly works. But it’s not actually a good layout at all. If you already learned QWERTY, well, too bad. You might consider re-learning touch typing with a different layout, but there is less gain to it than a lot of people would say is worth the effort. I was a QWERTZ (German standard keyboard) typist and went through the ordeal of re-learning. It took time. It definitely was hard. And it required quite a bit of resolve. However it means that I can type without pain in my wrists again and that alone was well worth it. In fact it feels a lot better so that it’s more fun to type, too.

Would I recommend getting used to an alternative layout if you already know a standard one? It depends. I’d certainly not recommend it to everybody. If you are typing a lot and you feel that you’re young enough to benefit from the change, then go for it! Otherwise stick to what you already know and do well.

But if you’re just planning to learn touch typing, please look at what ergonomic layouts are available for your language. People whose native language is English might want to take a look at Colemak or Dvorak. For the German-speaking I can really recommend NEO¬≤ (and also for people who need a lot of special characters and e.g. the whole Greek alphabet in both upper and lower case). Just learn it properly (i.e. not QWERTY!) right from the start.

If you’re at all interested in why QWERTY is not a good layout for us today and how it came to be, I recommend the DvorakZine. They offer a PDF that explains it all with a very nice and informative comic.

2) Input devices

In case the previous paragraph was not for you (i.e. you don’t touch type and you don’t want to learn it either), you can skip this one, too. It doesn’t make sense to determine which keyboard and mouse is right for you and invest a nice sum of money if all you’re ever going to do is typing with a couple of fingers only anyway. You won’t profit from the ergonomic principles applied to the device design. Get anything that you’re comfortable with and be happy that you saved the expenses.

If you do touch type however, consider your options. A lot of people swear by their mechanical switches. Especially if you’re a fast typist do yourself a favor and think about using ergonomic devices. Yes, they can be expensive. And yes, it takes time to do research, too. But it might be well worth it. Here’s why:

It’s an important investment not so much in further increasing your speed (there may or may not be a small improvement there, too) but in maintaining your health and thus keeping up with the efficiency that you reached. Fast typing puts considerable strain on your fingers and wrists. I knew this for a long time but chose to ignore it with a “surely won’t hit me” attitude. But then it did. So let me assure you that RSI and the like can cause serious pain. I’m not a very thin-skinned guy, but there were days the pain was strong enough to be a real problem, forcing me to wear some kind of splint.

Today with my ergonomic keyboards (one at work and one at home) I can type mostly without problems again. Still I encourage you to be smarter than I was and don’t get there in the first place. It’s not cool to have your small child on your arms and with the constant fear of dropping it because you cannot completely trust your wrists after a busy day…

I do not regret the investment for the Truly ergonomic keyboard (the old one) and I’m looking forward to try out the new split Cleave model as it becomes available. There are cheaper and more expensive devices. If you’re looking a the cheap ones (like those from Microsoft that I used before): It’s probably better than nothing. But I have to add that it didn’t really help me that much.

You probably wouldn’t consider the very expansive ones before you’re having problems with your hands. Just take a look at e.g. the Maltron 3D – it comes at the cost you might even get a very old car for (depending on where in the world you are)! I’d be curious to try one out, but I don’t have that much money to spare and chances are that I’d still like my TEK better anyways.

Are you working a lot with the mouse? You’d probably be shocked by how many kilometers you move the mouse each week (there are programs that record statistics like that). For mice there are so-called vertical mice that are meant to be ergonomic alternatives to the common devices. There are some cheap ones available. One that I tried out had some flaws that cheap mice have (low precision, awkward buttons, etc.) but was ok to try out working with a vertical model.

At work I’ve been using a more expensive model for a few months now. It works pretty well and I’m happy with it. At home I’m having a more common horizontal model and won’t buy a vertical one. Why? Because I’m actually trying to eliminate the mouse from my workflow as much as possible (more on that in the following parts of this article) and thus it doesn’t make that much of a difference.

Whatever you decide on keyboards and mice – it’s important to know that there are ergonomic alternatives.

3) Multi monitor

In most workflows you’ll have multiple applications open and in general it makes sense to keep an overview and avoid unnecessary switching between your programs: Minimizing your IDE to go to the browser so you can look something up works, but it’s not ideal. If you’re working on a somewhat complex problem, it would also certainly help to have both your IDE and your browser visible at the same time.

Of course you can resize both applications so that you’ll have them on the screen at the same time even if you use just one monitor. While that generally works, it’s extra work (which will sum up), you won’t see as much as if you had the applications in a full-screen window (probably not all of the relevant code in your IDE will fit, forcing you to scroll). Let’s be honest: You can work around this situation, but a workaround is not a solution.

You might think that one monitor totally suffices. I thought so, too. Then I got two at work and became used to them in just hours. Now I wouldn’t want to go back for serious work. At home most machines have still just one monitor, but my main workstation totally needs to have two. I suggest that you try out working with two monitors a couple of days. If you find that you can go back to using a single one and don’t miss anything, that’s fine of course. But chances are that you don’t actually want to.

There are people who say you should use three or even more monitors. There certainly are workloads where that makes a lot of sense. Workspace on the screen is precious and more is generally good. But there’s the point when you just have enough and adding more will not be beneficial anymore. In fact adding more is becoming less and less useful the more you are approaching that point. Adding a second monitor to your setup is revolutionary. Adding a third one is not likely to make as much of a difference already. And while some people have special requirements that mean they can make good use of even more screens, that really is not too common.

An example: As a sysadmin I need to keep an eye on our monitoring system so that I can react quickly if something happens. There’s a lot of information in there and even though it’s color coded, making more things fit on the screen is a win. I also frequently need our communication channel, so I usually have the monitoring system and the IRC client open on my second screen. The other is for terminal emulators, a web browser and stuff like that. By making good use of my window manager’s (more on that important topic in a future part) virtual screens I get along pretty well. Still I often have to send something over to the second screen for a while, covering the monitoring system. While that’s not a big problem (I’ll get an on-screen notification about critical events, anyway, but I won’t see things in warning state) it is an example of a situation where a third monitor might make sense.

Try out working with dual-monitors – graphic cards that support two are nothing that uncommon. Before you try out three, think about what your daily tasks are. Chances are that the expenses and the effort required for that kind of setup are higher than what the additional monitor might be worth for you.

What’s next?

The next post will switch gears from hardware to software. But what’s with other hardware? Sure, there’s a lot that could be written about it, too. But this post is definitely long enough already and I wanted to focus on the most important things here.

Building a BSD home router (pt. 8): ZFS and jails

Previous parts of this series:

Part 1 (discussing why you want to build your own router and how to assemble the APU2),
Part 2 (some Unix history explanation of what a serial console is),
Part 3 (demonstrating serial access to the APU and covering firmware update),
Part 4 (installing pfSense),
Part 5 (installing OPNsense instead)
Part 6 (Comparison of pfSense and OPNsense)
Part 7 (Advanced installation of OPNsense)

Fixing swap

This is the last part of this series of building a BSD home router. In the previous article we did an advanced setup of OPNsense that works but is currently wasting valuable disk space. We also configured OPNsense for SSH access. Now let’s SSH in and su – to root and continue! Choose shell (menu point 8) so that we can have a look around.

# df -h
Filesystem           Size    Used   Avail Capacity  Mounted on
/dev/ufs/OPNsense    1.9G    909M    916M    50%    /
devfs                1.0K    1.0K      0B   100%    /dev
/dev/ada0s1b         991M    8.0K    912M     0%    /none
devfs                1.0K    1.0K      0B   100%    /var/dhcpd/dev

Uhm… ada0s1b is mounted on /none? Seriously? Let’s get rid of that real quick:

# umount /none

How did that happen? This leads to the question: What does our disklabel on slice 1 look like?

# gpart show ada0s1
=>      0  6290865  ada0s1  BSD  (3.0G)
        0       16          - free -  (8.0K)
       16  4194288       1  freebsd-ufs  (2.0G)
  4194304  2096561       2  freebsd-ufs  (1.0G)

There you have it. The second one is all wrong, it’s not meant to be UFS! We have to correct it to have proper swap space configured:

# gpart delete -i 2 ada0s1
ada0s1b deleted
# gpart add -t freebsd-swap ada0s1
ada0s1b added
# swapon /dev/ada0s1b
# swapinfo 
Device          1K-blocks     Used    Avail Capacity
/dev/ada0s1b      1048280        0  1048280     0%

That’s better. Now we need to adjust fstab to make this change persistent:

# vi /etc/fstab

Change the ada0s1b line like this:

/dev/ada0s1b		none		swap	sw		0	0

Ok, we have some swap now, but we’re wasting most of the disk space of our drive. Let’s address that one next!

Preparing the system for ZFS

In the installer we created a second slice (MBR partition) as a placeholder:

# gpart show ada0
=>      63  31277169  ada0  MBR  (15G)
        63   6290865     1  freebsd  [active]  (3.0G)
   6290928  24986304     2  !57  (12G)

Let’s delete it and create a second FreeBSD slice instead:

# gpart delete -i 2 ada0
ada0s2 deleted
# gpart add -t freebsd ada0
ada0s2 added

Now we need to create a disklabel inside and create a partition for ZFS:

# gpart create -s bsd ada0s2
ada0s2 created
# gpart add -t freebsd-zfs ada0s2
ada0s2a added

OPNsense does not load the ZFS kernel module by default. So let’s do that now and also notify the loader to always insert that ko during startup (we’re using loader.conf.local because OPNsense overwrites loader.conf during startup):

# kldload zfs
# echo zfs_load=\"YES\" >> /boot/loader.conf.local

Then we set the ashift. This tells ZFS to adjust to a 4k blocksize which is better for most of today’s drives use instead of 512 byte ones, even though a lot of them will lie to you and claim to have 512 byte sector size. But even on a drive that really has 512 byte sectors, using 4k is better than using 512 bytes on a 4k sector drive. You will only lose some space if you have a lot of very small files in this case. In the other case however, you will hurt performance badly. If you know your drive and you want to use another blocksize, look up how to do it. Otherwise just set the ashift like this:

# sysctl vfs.zfs.min_auto_ashift=12
vfs.zfs.min_auto_ashift: 9 -> 12

With that we’re good to go and create a pool and some datasets.

Pool creation

I’m calling my pool zdata but feel free to name yours whatever you like better. I also enable compression on the pool level and turn off atime:

zpool create -O compression=lz4 -O atime=off -O mountpoint=none zdata /dev/ada0s2a

Next is creating some basic datasets that won’t be used directly (hence forbidden to mount) but only serve as parents for other datasets:

# zfs create -o canmount=off -o mountpoint=none zdata/var
# zfs create -o canmount=off -o mountpoint=none zdata/usr

Let’s move the old log dir and create some new directories:

# mv /var/log /var/log.old
# mkdir /var/log
# mkdir /usr/ports

On with some more datasets:

# zfs create -o mountpoint=legacy zdata/var/log
# zfs create -o mountpoint=legacy zdata/usr/ports
# zfs create -o mountpoint=legacy zdata/usr/obj

To make the system use those we need to add them to the fstab:

# vi /etc/fstab

Add these lines to the file:

zdata/var/log		/var/log	zfs	rw		0	0
zdata/usr/ports		/usr/ports	zfs	rw		0	0
zdata/usr/obj		/usr/obj	zfs	rw		0	0

Once these additional lines are in place, the datasets can be mounted and the old logs transferred to their new place:

# mount -a
# mv /var/log.old/* /var/log/

The directory /var/log.old is no longer needed, but the system currently has some file descriptors open that prevent deleting it. Just rmdir after the next reboot. Speaking of which: It is now a good time to do updates (and change the firmware to the libressl-based one if you haven’t switched already).

BTW: Don’t try to put everything on ZFS! I made some experiments booting into single user mode and moving over /usr and /var. The results were… not pleasing. After doing some reading I found that while OPNsense works well with ZFS datasets, it’s startup process doesn’t cope with ZFS very well. Place its configuration on ZFS and you’re left with a partially defunct system (that doesn’t know its hostname and won’t start a lot of things that are needed).

Full ZFS support is already on the wish list for OPNsense. It looks like that won’t make it into 17.7, but I’m pretty sure that it will eventually be available, making root-on-ZFS installations possible. Yes, pfSense already has that feature in their betas for the upcoming version 2.4. And they even ditched the DragonFly installer and use the familiar BSDinstall which is really cool (dear OPNsense devs, please also take this step in the future, it would be greatly appreciated!).

Is this a good reason to switch to pfSense? It might, if for you this is the one killer feature and you are willing to let go of OPNsense’s many improvements. But there’s one big blocker: If you make the switch you don’t really need to read on. You won’t be able to create jails easily. Why? Because pfSense heavily customizes FreeBSD. So heavily in fact that you cannot even use the ports tree by default! And that is truly a rather sad state of affairs. Sure, a lot of pfSense users actually use MacOS or even Windows and only want to ever interact with the GUI. BSD means nothing to them at all. But if you’re a FreeBSD user it’s pretty annoying if things simply don’t work (and OPNsense shows that there’s no real need to screw things up as much as pfSense does it).

Ports and jails

The OPNsense team provides packages for OPNsense that you can simply install via pkg. However they currently offer only 368 packages, so chances are that you want something that is not there. The FreeBSD ports tree on the other hand means that over 27,000 programs are easily available for you! So since OPNsense is based on FreeBSD (and tries to remain close to it) this is really an option.

On FreeBSD you’d probably use portsnap to get a snapshot of the current ports tree. This won’t work in our case since OPNsense doesn’t have that tool. The other common way on FreeBSD is to use svnlite and checkout the ports tree from the Subversion repo. Again OPNsense doesn’t provide that tool. And it also doesn’t package the full SVN.

So what can we do to acquire the ports tree? OPNsense does provide a git package and the FreeBSD project offers a git mirror of the SVN repositories. But wait a second! OPNsense works together with the HardenedBSD team and they have their own ports tree (based on the vanilla FreeBSD one with some additions). The whole ports tree is pretty big, but we don’t really want (or need) the whole history. Just what various version control systems call “head”, “tip”, “leaf”, … For git we can achieve this setting the “depth” to 1:

# pkg install git
# git clone --depth=1 https://github.com/HardenedBSD/hardenedbsd-ports.git /usr/ports

FreeBSD ships with OpenSSL in base and a lot of ports expect to link against that. We’re however using LibreSSL and so we have to tell the build system to use that by making an entry in make.conf:

# echo DEFAULT_VERSIONS+=ssl=libressl >> /etc/make.conf

If – for whatever reason – you decided to stick to the OpenSSL firmware, you still need to edit make.conf. This is because OPNsense uses OpenSSL from ports which is usually newer than the version from base (that cannot be upgraded between releases for ABI stability reasons). Use ssl=openssl in that case.

The next step is optional, but I recommend installing a tool for dealing with ports. My example is a pretty light-weight port but maybe you want to build something more demanding. Especially in those cases a ports management tool comes in very handy. I suggest portmaster which is extremely light-weight itself:

# make -C /usr/ports/ports-mgmt/portmaster install clean

Once you have it installed, you can install the jail management tool. Yes, I know that I’ve written about py3-iocage a while ago, but that comes with a lot of dependencies and doesn’t provide enough of an advantage over the purely shell based iocell fork. For that reason I would simply go with that one in this case:

# portmaster sysutils/iocell

Alright! Now you have iocage installed and can start creating jails. What services would you want to jail on a small router box that is always on? Think about it for a moment. There are many great possibilities (I’ll likely write another article soon about what I have in mind right now).

Looking back – and forward

What have we accomplished in this series? I now have a frugal little router on my desk that is quietly doing its work. So far it’s just an additional machine between my network and the modem/router box from my ISP. Taking a break from topics directly related to the actual router, I’ll setup some jails (and NAT) next. But then there is a lot more to look into: How to do proper firewalling? What about traffic shaping? How to configure logging? Also VPN and VoIP come to mind as well as NTP, a DNS cache or even vLANs or intrusion detection.

OPNsense places so many tools within reach of your hands. You only have to grab one of them at a time and learn to use it. That’s what I intend to do. And then, some point in the future, equipped with much more solid networking knowledge, I’ll try to replace that box I got from my ISP with my own modem, too. But excuse me now, I have some reading to do and configurations to break and fix again.

Building a BSD home router (pt. 7): Advanced OPNsense installation

Previous parts of this series:

Part 1 (discussing why you want to build your own router and how to assemble the APU2),
Part 2 (some Unix history explanation of what a serial console is),
Part 3 (demonstrating serial access to the APU and covering firmware update),
Part 4 (installing pfSense),
Part 5 (installing OPNsense instead) and
Part 6 (Comparison of pfSense and OPNsense)

Revisiting the initial question

In the first post I asked the question “Why would you want to build your own router?” and the answer was “because the stock ones are known to totally suck”. I have since stumbled across this news: Mcafee claims: Every router in the US is compromised. Now Mcafee is a rather flamboyant personality and every is a pretty strong statement. But I’m not such a nit-picker and in general he’s definitely right. If you have a couple of minutes, read the article and/or watch the short Youtube interview that it has embedded.

If you care about things like privacy at all, we’re living in a nightmare already and things keep getting worse. What I have blogged about in this series of posts so far is not really solving any problem. It’s just a first step to take back your network. Have you built your own router, too, or are you planning to do so? Just assembling it and installing a firewall OS on it won’t do the trick. As a next step you have to learn the basics of networking and firewalling so you can configure your box according to your needs. And even then you have just put your own router behind the modem/router box from your ISP and not replaced that. I’d like to go further and get my own modem, too. But that step requires a lot more reading before I will even attempt to do it.

Manual installation

However this article is about doing a more advanced OPNsense installation that leaves room for customizing things. Let’s get to it!

OPNsense Installer: Manual installation

In the installer select “manual installation” obviously. This will lead you through a couple of dialog windows that let you customize your partitioning etc.

OPNsense Installer: Format the disk?

It seems like OPNsense can be installed on an existing filesystem. There might be people who would want that feature but I don’t. I definitely prefer to start fresh as a newly installed OS should be in a clean state in my opinion.

OPNsense installer: Geometry confirmation

The installer then gives you the option to change the disk geometry. You almost certainly don’t want to do this. If you do need to, you have a strange disk, are aware of its quirks and know geometry matters good enough that you definitely don’t need my advice on it.

OPNsense installer: Slice disk?

Next you are asked if you want to slice (OPNsense uses the term “partition” to describe MBR partitions which is fine since that’s what non-BSD people usually call it). I don’t expect to be dual-booting my box or anything, so I could go with just one slice. However I might install and try out some other versions (or take a look at pfSense again when 2.4 is officially out or even something like OpenWRT, just to take a look at it). For that reason I create two slices so I can keep my OS on one and my data on the other.

OPNsense installer: Disk slicing

I created a FreeBSD slice and one of type Plan9. No, I’m not going to put Plan9 on there. It will be erased and re-purposed anyway. But the installer has this option and Plan9 is cool. 3 GB for OPNsense should be enough and I give the rest to the future data slice.

OPNsense installer: Slice alignment

For the advanced installation we’re unfortunately stuck with installing on the MBR partitioning scheme. That means (for compatibility’s sake) the system enforces the old CHS (Cylinder, Head, Sector) addressing limitations which are almost completely irrelevant today, but meh. The most annoying consequence of this is that “partitions have to end on a cylinder boundary”. If you don’t know what that means: It’s related to the physical geometry of spinning drives that has been of high importance in the olde days(tm) and still haunt us today because operating systems are used to work with it (even though geometry parameters have been lies and lies for decades now and SSDs don’t have spinning parts but claim to have them to make the OS happy…). To comply with this, choose to grow or shrink your slice by a couple of sectors.

OPNsense installer: No bootblock installation

If you want to dual-boot (or multi-boot) your box, make sure to install a boot manager now. I don’t anticipate to install more than one OS on it at the same time and so I skip this. Oh, and please don’t ask me what “packet mode” is! I tried to research it, but all that I found boils down to “if you have problems, try with/without it”. I couldn’t really find anything about what that actually does (at least not in a reasonable amount of time). If you know: Please leave me a comment!

OPNsense installer: Slice selection

Next is selecting which slice to install to. Why, the FreeBSD one, of course!

OPNsense installer: Adding disklabel partitions

Finally the slice needs to be partitioned (or sub-partitioned if you regard the slices as “partitions”!). This means that BSD disklabels are created inside the MBR slice to allow for multiple partitions. For the setup that I have in mind, two partitions suffice: One for / and the other for SWAP space. For whatever reason the installer does not directly allow to assign SWAP, so I allocate 2 GB for the root partition and the rest to a second partition that has no mountpoint. That’s it, the installation can start now.

SSH access

Once the installation is complete, follow the steps that I wrote about in the article about the simple installation.

Got the interfaces assigned and the setup wizard run? Good. OPNsense can be administered purely through the Web GUI. Howver if you’re like me, you really prefer some means of direct console access. Sure, we have that over the serial console. While that’s fine for the installation, it’s a bit cumbersome for daily use. Fortunately there’s a better way: Let’s just enable SSH access!

OPNsense Web GUI: Creating a user

First stop: Creating a user (you wouldn’t want to SSH in as root, do you? Do you?! Do this on a production machine like never). One thing is important here: Make your new user part of the “admin” group or else it won’t be terribly useful to you. Also use SSH keys instead of passwords. If you haven’t ever used keys, set a couple of minutes aside to do a little reading about what they are. They are much more secure than passwords and you definitely want to use them even if you don’t know that just yet (I recommend the article on SSH keys over at the Arch Linux wiki. Unless you’re using the original OpenBSD OpenSSH, we’re all using the same version of OpenSSH-portable anyway). You must also check the “use scrambled password” checkbox because OPNsense won’t let you get away with an empty password.

OPNsense Web GUI: Enabling SSH

Then OpenSSH needs to be activated. If – for whatever reason – you cannot use keys, you have to enable the “permit password login” option. Try to avoid that, though. And don’t check the “permit root user login” however convenient it might be!

SSH login to the OPNsense box

That’s it, you can now log into your box using SSH. Use su – and the root PW to become root. OPNsense will then display the nice menu that you already know from connecting via serial.

What’s next?

Right now we have a lot of disk space wasted and there’s other things wrong, too. So after the installation there’s some more work to do, some packages to install, filesystems to create, etc. I originally intended to stuff more into this post but it’s certainly long enough already. See you in part 8, the last part of the series!

Building a BSD home router (pt. 6): pfSense vs. OPNsense

Part 1 of this article series was about why you want to build your own router, and how to assemble the APU2 that I chose as the hardware to build it from. Part 2 gave some Unix history and explained what a serial console is. Part 3 demonstrated serial access to the APU and showed how to update its firmware. Part 4 detailed installing pfSense, while the previous one did the same with OPNsense.

A little overview: In this post I will give you some background information, compare the appearance / usability of both products and then take a look at some special features before giving a conclusion.

pfSense vs. OPNsense: Who wins?

This article is about comparing both products and helping you to make a decision. It is not terribly in-depth, because that task would require its own series of articles (and a lot more free time for me to dig much deeper into the topic). But still there’s a lot you may want to know to get a first impression on which one you should probably choose. If you do some more research and write about it, please let me know and I will happily link to your work!

I want to point out one thing right at the beginning: Both products are good firewall solutions with a heck of a lot of extras. If you have the same goal as I have (building a home router), either will do absolutely fine. That does of course not mean that your choice doesn’t matter at all. You can definitely benefit from thinking about it before making a decision. But even making the “wrong” decision doesn’t mean that it will be horribly wrong. There are a couple of differences and maybe they are important to you. But chances are that both products would completely satisfy your needs.

Heritage

Sometimes it’s helpful to ask the old question: “Where do we come from?” While this question is usually a philosophical one, in our case it helps to shed some light on our topic. If you do a little reading on the net, you will soon find that pfSense and OPNsense do not like each other much. In fact it’s probably safe to state that they are more or less hostile to each other. OPNsense is a fork of pfSense. Obviously not a friendly fork.

Some pfSense enthusiasts have been spreading information about OPNsense which suggests that new team has no idea what they are doing. They are said to frequently break important things and that the whole project is actually quite laughable. Or to put it short: You really should not waste your time with it and stick to the original. Having liked pfSense for years, I would have believed that, even though you should listen to the other side, too, before doing so. But listening to both sides takes time and effort – both of which were rather limited when I briefly looked into the whole clamor in mid 2015.

Eventually it was the plain hatred of one person who appeared to really have no life, that made me look at what the other project would say. This strange guy popped up in every single pfSense vs. OPNsense discussion and threw so much dirt at OPNsense that I could not help but pity that person. The fact that pfSense (despite obviously being completely dependent on pf, a technology that came from OpenBSD) has a rather bad name with a lot of OpenBSD people for behaving poorly in the past, didn’t help to regain my faith in it, either.

According to OPNsense, they were not happy with the code-quality of pfSense. They didn’t like the fact that the whole Web GUI ran as root (ouch!) and wanted to do privilege separation (which is actively work in progress as I was told). Also there were licensing issues when Netgate acquired pfSense and a bunch of other things. Deciso, a company based in the Netherlands, had been a sponsor of pfSense for years but felt that the whole project was going in the wrong direction after Netgate took a couple of actions. So they decided to fund a fork instead.

Who is right? There’s probably some truth to both versions. OPNsense has followed a rapid development style, bringing in lots of new features and even making some rather drastic changes. It’s true that especially in the beginning there were some problems due to that. But it’s also true that they were quick to react to those. One thing that is not true (or at least not the whole truth, if you will) is that pfSense is the original and OPNsense is a cheap rip-off! What’s the whole truth then?

Once upon a time… in 2003 there was a new firewall OS called m0n0wall. Manuel Kasper had built it on a stripped down version of FreeBSD. There had been small firewalls before, but Kasper’s innovation was to put a Web GUI on top of it so that the firewall’s settings could be controlled from the browser! It did not take long and m0n0wall took the world by storm. However Kasper’s project focused on embedded hardware. So only a while later a fork was created which geared towards more powerful hardware. The fork’s name? You’ve guessed it: pfSense. In 2015 Manuel Kasper officially ended the m0n0wall project (because recent versions of FreeBSD had been grown too big to be easily usable for what he did with it in the past). And guess what he did: He gave his official blessing and recommends to migrate to and support OPNsense!

Appearance

Knowing some background is nice, but what do both products feel like? The first major difference between the two is what they look like. This is pfSense’s main dashboard:

The pfSense dashboard

Compare it to OPNsense’s version of the dashboard:

The OPNsense dashboard

As you can see, OPNsense did a lot to provide the user a much more modern GUI. Both dashboards are customizable but it’s hard to argue that OPNsense’s is not superior. But to be fair: pfSense is working on a GUI overhaul as well.

Let’s compare a couple of the menus. This is pfSense’s “System” menu:

pfSense: “System” menu

And here’s the one from OPNsense:


OPNsense: “System” menu

While pfSense uses pull-down menus at the top, OPNsense has a navigation bar to the left. As you can see, both have not that much in common. This is because OPNsense did not only redesign the GUI but also re-arranged which options go where. I find the new arrangement more logical (e.g. with pfSense logout is in “System” but halt is in “Diagnostics”). But that’s definitely a matter of taste.

Here’s what the “Services” menu from pfSense looks like:


pfSense: “Services” menu

And here’s the corresponding one from OPNsense:


OPNsense: “Services” menu

This time there seems to be quite a bit more consensus about what counts as a service. But still OPNsense looks more like a cleaned up version.

Another example is the “Diagnostics” menu in pfSense:


pfSense: “Diagnostics” menu

There’s no direct equivalent with OPNsense. In the “Service” menu above you can see that there is a “Diagnostics” entry. The same goes for “System”, “Interfaces”, “Firewall”, etc.

And now for the heart of the whole thing – here’s pfSense’s (default!) WAN firewall rule settings:

pfSense: WAN firewall rules

Compare that to the same thing from OPNsense:

OPNsense: WAN firewall rules

This is where it shows that both products do have a lot in common: What we can see here is basically the same thing. Again OPNsense simply has the more modern interface.

To end the visual comparison let’s look at the LAN firewall rules from pfSense, too:

pfSense: LAN firewall rules

And here’s the LAN rules from OPNsense:

OPNsense: LAN firewall rules

No surprise here: It’s all very similar just with interface improvements on OPNsense’s side.

Technical

So far it’s mostly a matter of taste. But now on to the technical points. This is where OPNsense shines (which is no wonder since it’s developing a lot faster). For example OPNsense is already based on FreeBSD 11.0 whereas pfSense is 10.3-based. However there’s already beta versions for the upcoming pfSense 2.4 which are also based on 11.0 and feature many more improvements.

One major difference between the two is that pfSense heavily customized FreeBSD while OPNsense believes in the opposite and tries to be as close to mainline FreeBSD, just adding packages on the top of it. I like that approach better but again that’s probably a matter of taste as well.

The most important thing for me is that OPNsense entered into a partnership with the HardenedBSD project. This resulted in OPNsense being able to change the crypto framework used! For me this is the one killer feature. Give me the option to rip out OpenSSL and use LibreSSL instead and I’m sold! However that’s not even all, yet.

OPNsense: Selecting alternative firmware

OPNsense got HardenedBSD’s ASLR (Address Space Layout Randomization) implementation and the most recent addition is the introduction of packages compiled with SafeStack. This is what the most current update notice looks like (it didn’t fit completely on the screen):

OPNsense: Example of an update notice

If you ask me, hardening your router (especially if it should happen to be promoted to be your border router eventually) makes a lot of sense.

There would be much more to write here, but if you’re interested in that, you will probably have to read some of the recent change notes yourself.

Conclusion

OPNsense and pfSense are quite similar in their core functionality. When should you choose which one? Have a look at the pros and cons of each one and decide for yourself!

pfSense:

  • + Better known brand (= more material about it on the net)
  • + 2.4 will feature BSDinstall including root-on-ZFS
  • +/- Slower development
  • Missing a lot of innovations / security enhancements compared to OPNsense
  • Has been acting quite unfriendly in the past
  • Rather intransparent future direction

OPNsense

  • + Already based on FreeBSD 11.0
  • + Nice new GUI, sensible rearrangement of items
  • + Pushing for priv. sep., uses current technology (PHP 7, phalcon 3, …)
  • + Partnership with HardenedBSD (optional use of LibreSSL, SafeStack, …!)
  • + Closer to unaltered FreeBSD
  • + Bootstrap script to turn a vanilla FreeBSD installation into OPNsense
  • + Multiple languages supported
  • + Public roadmaps for future versions
  • + Official blessings from m0n0wall founder Manuel Kasper
  • +/- Fast-paced development, rapid update policy
  • (Currently) less well-known than its competitor
  • (Currently) installer problems with some USB3 devices
  • Not currently able to install root-on-ZFS

What’s next?

The next article of this series will give an example of an advanced install of OPNsense that lets you use the APU for more than just a router!

Building a BSD home router (pt. 5): Installing OPNsense

Part 1 of this article series was about why you want to build your own router, and how to assemble the APU2 that I chose as the hardware to build this on. Part 2 gave some Unix history and explained what a serial console is. Part 3 demonstrated serial access to the APU and showed how to update its firmware. The previous article detailed installing pfSense.

This post will show how to install OPNsense, a great alternative to pfSense.

Preparation

OPNsense was forked from pfSense (more on than in the next post) and thus you will find lots of similarities if you have read the post on installing pfSense. The OPNsense team decided to move forward more quickly and did lots of interesting but invasive changes. One strong point for example is that it is already based on FreeBSD 11.0. There is one drawback to this, however: a problem with the XHCI (USB3) driver can lead to the installation media not being able to mount the filesystem and boot up. This makes installing OPNsense a little bit more complicated since the APU2 only has UBS3 ports.

Well, the board does have an internal USB2 controller, too. Therefore I suggest getting a cable that allows connecting USB devices to it. If this is not for you, take a look at the end of the post, I’ve prepared a section “alternative installation methods” there.

First download an image (select amd64 + serial). Then dd it onto an unused memstick and prepare the serial connection (take a look at the previous posts if you need help with dd’ing or attaching the serial console).

Open APU2 box with serial connection and memstick attached to the internal USB2 controller

As you can see, I’ve attached a memstick with OPNsense via USB2 and made a serial connection. That way the installation works just fine.

Step 1: Installation

Hit F10 to go to the boot menu as soon as SeaBIOS offers it.

Boot menu to select which device to boot off of

Since we’ve attached the memstick over USB2, the internal drive would take precedence over it in the default boot order. So in this case I have to select 2 to boot off of the memstick.

The OPNsense boot loader

The OPNsense boot loader looks fine. If you’re installing 17.1 using USB2 you don’t need to do anything here.

Nice feature: Early configuration importer

One notable difference from pfSense is the early configuration importer. If you have a saved configuration XML file, you can put e.g. a UFS2 filesystem on a memstick, create a directory conf on it and copy config.xml there. That makes it available in the importer.

Interface assignment

Then you have the option to assign roles to your interfaces (like WAN and LAN).

Logging into the installer

OPNsense gives you the choice to start the installer or to use a live system. Log in as user installer to perform an installation or as root in the other case. The password for both users is opnsense.

Greeting screen of the installer on the serial console

The OPNsense installer is black and white only when using the console. But that’s fine. The installer greets you with the welcome message.

Console configuration menu

The next screen lets you customize the console. You probably don’t need to do that.

Selecting the installation type

Then you need to select the installation type. You could do advanced partitioning here or setup a softraid (gmirror). We’re going with the simple installation for this post.

Choosing the drive to install on

Now you need to choose where to install to. The mSATA drive is ada0 whereas the memstick is da0.

Selecting the partition scheme to use

OPNsense also lets you choose which partition scheme to use. In case of our router this is not terribly important, especially not with our sample installation that puts everything in one partition. But since stone age is over, you might as well choose GPT anyway.

Progress bar for the installation

While the progress meter was broken with pfSense, this has obviously been fixed for OPNsense. Not that you should reinstall all that often, but still…

Installation done: Reboot!

Once the installation is finished, you of course want to reboot to your new system.

Displaying some information before rebooting

Before rebooting, OPNsense tells you how to access the Web GUI. However the IP address that it uses by default is already taken by my ISP’s modem/router box. We’re going to change that next.

Step 2: Text mode configuration

When the system has started up, you are prompted to log in. This is the default behavior which can be changed to allow unprotected login over the console like with pfSense. But in general I like that bit of extra security.

OPNsense’s text-mode configuration menu

The text-mode configuration menu looks much like that of pfSense.

Configuring the LAN interface

And the interface configuration works right the same.

Setting up DHCP on the LAN interface

As does the DHCP configuration.

Logging out and disconnecting the serial console

Since OPNsense required a login, you can also log out when you’re done. Now disconnect the serial console – we’re done with it.

Step 3: Web GUI configuration

Just like pfSense, OPNsense offers a nice Web GUI to configure all the settings. Fire up your browser on a PC that is in the same subnet (or got its IP address via DHCP from the new router) and enter the router’s LAN IP address in the URL bar.

Self-signed certificate warning

OPNsense uses https to create a secure connection, too. Of course a self-signed certificate is used which is not trusted by my Firefox. Therefore a permanent exception needs to be made.

OPNsense Web GUI login screen

Once you have confirmed the exception, you will see the login screen. Log in as root with the password opnsense.

The configuration wizard

On the first login you will be greeted by the configuration wizard. It will present you about the same choices as pfSense does (without the advertizing of the commercial version, of course).

Configuring general settings

First it’s some general information like hostname and DNS. What OPNsense offers over pfSense is i18n options: Chances are that you can configure the Web GUI to speak your language! That’s pretty nice.

Configuring time-server settings

Time server settings are just like those from pfSense.

Configuring the WAN interface

WAN configuration offers you a lot of options. Take a close look at those. Fortunately you very likely don’t need most of what is there.

Configuring the LAN interface

Same thing for the LAN configuration: You know that from pfSense.

Setting a new password for the Web GUI

Also with the password changing part there’s no surprise here.

All done. Reload the config!

That’s it. Reload the config now and you’re done with the wizard. OPNsense now has a basic configuration and is ready to be used.

Alternative installation methods

OK, you don’t have a cable to connect to the USB2 pins but you want OPNsense? There are several things that you can try. I’ve documented my attempts (including several solutions) on the OPNsense forums in case anybody needs them.

Here are a few things that you can try:

  • Install from SD card (I didn’t try that but it should indeed work)
  • Install 16.7 from USB3 with increase boot_delay and then update
  • Install 17.1 using a USB cdrom, manually enabling the console and importing a pre-made configuration

Should you install 16.7 using a USB3 port, press ESC before the loader countdown runs out. This will drop you to the loader prompt. Then enter the following:

set kern.cam.boot_delay=10000
boot

That did the trick and made the system boot up for me. The actual installation is quite similar to what I covered above.

You could also use a USB cdrom to boot the installation – of course use the OPNsense cdrom ISO in this case! However the cdrom image does not have the serial console enabled by default. So escape to the loader prompt, set some variables to enable the serial console and boot:

set boot_multicons=YES
set boot_serial=YES
set comconsole_speed=115200
set console=comconsole,vidconsole

This will work, too. But there’s one little problem with that: The TTYs are configured on their own using a configuration file – and they are not ready for serial connection! Since this is a CD, we cannot really do much about that. What we can do, however, is using the configuration importer. I will upload a basic configuration xml and add it to this post when I next install a clean OPNsense.

What’s next?

The next post will be pfSense vs. OPNsense! It will discuss some of the notable differences and when to use which one.

Building a BSD home router (pt. 4): Installing pfSense

Part 1 of this article series was about why you want to build your own router, and how to assemble the APU2 that I chose as the hardware to build this on. Part 2 gave some Unix history and explained what a serial console is. Part 3 demonstrated serial access to the APU and showed how to update its firmware.

This post is about the serial installation of pfSense, one of two FreeBSD-based router/firewall operating systems that we’re going to explore in this series (the other being OPNsense). As pfSense is the older and more established product, we’re beginning with that one.

Preparation

We’re just doing the installation here. A closer look at using pfSense or a comparison with OPNsense will be another post. Getting pfSense up and running is really easy, even when you’re using the serial console. The first step is the actual installation. In a second step you need to configure the LAN interface and then you can use the WebGUI to do the final setup.

The first thing to do, however, is getting preparing an installation medium. Head over to pfSense’s Download site. What you want is an install image for amd64. Then select USB Memstick Installer which let’s you choose the console type – obviously get the serial one!

Then get a USB stick that you can spare and dd the image on it. Once you have that ready, plug it into the APU. Next attach the serial cable to your APU and to another computer. Then connect to the console (how to do that was described in the previous post). Now power on the APU.

Step 1: Installation

Even if there’s already an OS installed on your mSATA drive, the memstick should take precedence when it comes to boot order. So you can probably just wait until the installer comes up.

pfSense’s loader menu: screwed up over the serial console…

Don’t be scared when you see garbage displayed on the screen. This is just the bootloader that’s screwed up badly when used over a serial connection (they’ve already fixed that in the beta version for the upcoming pfSense 2.4). Either just wait 10 seconds for it to boot automatically or press enter to boot right now (if you need any other options, you might want to get an ISO for pfSense, too, and test it in a VM or get a VGA image, put that on a stick and try it out on hardware that provides a local console over a screen and keyboard).

…but once the kernel loads, text is fine

As you can see, it’s only the loader. As soon as the kernel takes over, the text is displayed correctly. That means you can actually read the messages in case anything goes wrong here. If you don’t do anything, the installer will eventually come up automatically.

First screen of the installer

In the first screen of the installer you can configure the console. Most likely the defaults will be fine, though.

Selecting the installation method

Then you need to choose the installation method. We will do a quick installation but you could also do a custom installation or setup gmirror (mirrored software RAID).

The usual “this will erase your data” warning

Since installing pfSense means destroying any data that might currently be on the drive, the installer warns you that it will erase it.

Installation progress bar

If you confirmed the warning, the actual installation starts (but the progress meter is kind of useless as it seems… It remained at 5% for a while and then jumped to 100% for me).

Kernel selection

The next thing to do is to select the right kernel. Since our APU2 is a headless device, make sure that you select the embedded kernel! Otherwise you won’t be able to use the serial console with it.

Another progress bar

After the kernel is installed, the installer runs a script to do some final tasks.

Reboot message

When all is done, it’s time to reboot the system.

pfSense rebooting after installation

Just before it reboots, pfSense prints some important information on the screen, telling how to log into the WebGUI. Remove the memstick now or the APU will boot off of it once more an you’ll just see the installer again.

Step 2: Text mode configuration

The OS has been successfully installed, but leave your serial console attached for now.

pfSense’s text mode management menu

Once the system has booted, you will see the management menu. It offers a lot of tools including going to a shell (option 8) and doing everything you like. We want to configure the IP address for our LAN interface (option 2):

Configuring the LAN interface

I’m assigning 192.168.2.1 since my modem/router (yes, I’m not replacing it just yet and will operate the new router between that box and my actual network for now) has already taken 192.168.1.1. It’s not like I need a full /24 subnet for my network, but I go with that subnet mask for now.

Configuring DHCP for the LAN interface

Since I intend to use DHCP for my network, I enable a DHCP server for the LAN interface. The range of DHCP addresses that I use here is just an example for this test installation. I will cut it down to about 10 when I do my final setup. The reserved addresses before the DHCP range serve a purpose, though – more on that in a separate future post.

Back at the menu

As soon as everything is ready, you can now end the serial connection and remove the cable. We have a valid IP address on the LAN interface now after all.

Step 3: WebGUI configuration

So now we can access the WebGUI simply by entering the IP address in the URL bar of any browser. Of course the computer that runs the browser have an IP address that is on the same subnet. So you might want to change your address if that is not the case – or fire up the dhclient, it should get an address in the range that you specified (or simply reboot if your computer is configured for DHCP).

Self-signed certification warning

It’s a good thing that pfSense uses TLS so you can access the router securely via https. However the certificate it uses is self-signed and thus unknown to your browser which will display a warning. That doesn’t mean that it’s useless. In our case it’s just necessary to create an exception to accept that cert permanently.

Logging into pfSense’s WebGUI

You’ll then see the login screen. Use the username admin and the password pfsense to log in.

Running the configuration wizard

Once you’re logged in, pfSense suggests that you run the configuration wizard – and that makes sense.

A little advertising for pfSense Gold

The first screen of the wizard is an advertisement for the commercial version of pfSense called pfSense gold. If you are a company looking for more than the free “Community Edition” of pfSense will give you, have a look at this service. Maybe it’s for you.

General information configuration

First you configure some general settings like the hostname, domain, etc.

Time Server configuration

Next is the configuration of the time zone and NTP daemon.

WAN configuration

Then the WAN interface needs to be configured. There are a lot of settings there and very likely you don’t need all of them.

LAN configuration

After that comes the LAN interface. Here you can only configure the IP address and subnet mask (which we already did in text mode).

Changing the password for the WebGUI

Finally we’re prompted to change the password which is a good idea of course. Even if the WebGUI is only accessible from the LAN interface by default, it’s a matter of principle.

Configuration done: Reload!

That’s it, the wizard is finished. Time to reload the configuration.

All done, pfSense is ready

We’re done here, pfSense is installed and the basic configuration has been applied. There’s another little advertising here which is legit for a free product, I guess. We’re going to take a look at the main WebGUI and its many, many options in another post.

What’s next?

The next blog post will detail the installation of OPNsense, another excellent option for your router.