FreeBSD router take 2 (pt. 4): Demoting my ISP’s router

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/bsd_router_take_2_pt4.gmi

Since I built my first OPNsense-based router, it had been a secondary router only. Its “WAN” port was connected to my ISP’s modem/router box which dealt with establishing the actual Internet connection and acting as the gateway and DHCP server for my OPNsense. In other words: It has only ever been a second line of defense for my LAN network behind it.

Also since I started building said first router of mine I had the goal of eventually replacing my ISP’s router with it instead of just adding a secondary device. In retrospect it still was the right choice to start with a first step like I did. It allowed me to play with OPNsense and get familiar with it while I did not fully depend on it working right all the time. It has been fun and the fact that OPNsense never broke for me when updating was a reassuring experience.

Promoting OPNsense to be my primary router

The following image shows what my network looked like up until a short while ago (red = my ISP’s property, blue = OPNsense, green = transparent networking devices):

Diagram of my network before the change

As you can see, my ISP’s modem/router was the device directly connected to my ISP’s line, forming my primary network behind it (LAN 1). Occasionally I’d plug a laptop or something into that directly. The APU running OPNsense was a secondary router behind which the network for my regular devices began (LAN 2). It meant that I could better tune packet filtering rules than the primary router allowed me. I could do proper QoS and other things. But it only was an additional device and didn’t really obsolete the one that I actually wanted to get rid of.

I did a bit of research and finally in early April I went ahead and made the switch. For my network OPNsense is in charge now! But I didn’t actually get rid of my ISP’s old box just yet. Why? Well… because of IP telephony. This is a topic of its own and I hadn’t been inclined to doing too many things at once. So I decided to try out if my phone would still work if I demoted my ISP’s router to be the secondary router and would let it manage telephony. I just set the device to operate in client mode, connected the IP phone, tried to call my parents – and was very relieved to find that it just worked!

So this is the new network diagram:

My network after the change

The APU is not technically capable of connecting with my ISP; I needed a DSL modem for that. The model that I chose operates in bridge mode so that it’s network-transparent but let’s OPNsense establish the connection. Directly connected to that additional device is the APU which now also handles a second network segment via its OPT1 interface: My old router is connected to that.

At the end of the day I have one more device involved but packets originating from my main computers no longer have to go through two routers. VoIP packets have to now, but at least the primary router is the box that I control, so that’s an improvement as well.

Choosing a DSL modem

After my research, I settled on the Zyxel VMG1312-B30A which is marketed as a “Wireless N VDSL2 4-port gateway with USB”. It’s an older device from 2012, but it’s still sold. While the specs don’t look very impressive today, I don’t care about which wireless standard it supports and such. I got it for another feature that it offers: Bridge mode.

Zyxel VMG1312-B30A DSL modem top view

If I were to simply replace my ISP’s device with this one, it’d even be a downgrade – the Zyxel makes an even worse router than what I had. But operating in bridged mode it simply terminates the DSL circuit of the telephone line and communicates using the DMT (Discrete Multitone Modulation) protocol with the ISP’s DSLAM (Digital Subscriber Line Access Multiplexer).

Zyxel VMG1312-B30A DSL back and bottom top view

Before settling on a modem, do some research on which technology is being used for DSL in your country! For my Zyxel device there are two separate variants of the same model: One for the so-called Annex-A and one for Annex-B. The former specifies “DSL over POTS” (= Plain Old Telephone Service) while the latter is for “DSL over ISDN”. Both devices are physically different, so be sure to get the right one (in Germany for example it’s Annex-B)! Annex-A uses the smaller RJ-11 jacks to connect to the ISP line while Annex-B uses the standard RJ-45 jacks that are also used for ethernet cables.

Taking a first look at the modem

By default the modem operates in router mode and has the IP 192.168.1.1/24 assigned. Configuration is accessible via a web UI on ports 80 and 443. The user is admin and the password 1234. Configure a workstation a static IP in the same subnet and connect it to the device, then login.

The modem has a standard overview page called “connection status” and four more sections that offer a menu each. First one is Network Setting.

“Network Setting” menu

The most important pages in that menu are Broadband, Wireless, Home Networking and Power Management.

Menu number two is called Security.

“Security” menu

This time there are two interesting pages: Firewall and MAC filter.

The third menu is “System Monitor”.

“System Monitor” menu

There you will find logs, the ARP table, the routing table and so on if you need it.

Finally there’s the “Maintenance” section.

“Maintenance” menu

The most important pages here are User Account, Remote MGMT, Firmware Upgrade and Configuration.

You should probably start by updating the firmware to the latest available version, but I’ll go through some of the pages here in order.

Modem configuration

First go to Network Setting -> Broadband. Set the device to operate in Bridge mode. Depending on what ISP you are using, you might need to have to set a VLAN tag for the connection to work. In my case choosing VLAN 7 is required. You might need to do some research or try out some possibilities.

Make the modem work in bridge mode

Then go to Network Setting -> Wireless. Do yourself a favor and just disable it. It will save you some power and offer additional security. If you need to update the firmware again or make another change, just physically connect a machine to it.

Turning off wireless access

Next is Network Setting -> Home Network. Turn off the DHCP server there. And if you’re paranoid, assign it a different IP – preferably in a different private address range.

Disabling the DHCP server

Lastly for the first section go to Network Setting -> Power Management. Here you can turn off everything that you don’t need. I only left the WAN port as well as one LAN port active and chose to unpower the rest.

Unpowering the LED and most ports

Next is Security -> Firewall. Since we’re not using Router mode, the firewall doesn’t make any sense. Off it goes.

Switching off the Firewall

For a bit of extra paranoia go to Security -> MAC Filter. Here you can choose to allow access to the modem only from certain NICs. If you consider doing this, make sure that you understand the consequences. Allow a minimal of two MAC addresses to not lose access if the respective NIC / machine should ever get damaged. If you only configure one, make sure to at least write it down and deposit it somewhere safe in case you need to spoof it. Otherwise you’ll have to factory-reset the modem when you managed to lock yourself out.

MAC Filter interface

Definitely go to Maintenance -> User Account and change the default password to something stronger.

User Account settings

Pay Maintenance -> Remote MGMT a visit. Turn off everything that you don’t need. You definitely don’t want Telnet, FTP or plain HTTP. Chances are that you don’t want SNMP either (if you do want to have it you know why). Disable ping if that makes you more feel better. And when it comes to SSH, here’s the reason I turned it off:

Unable to negotiate with [IP ADDRESS] port 22: no matching exchange method found. Their offer: diffie-hellman-group-sha1

This means that they ship a version of OpenSSH from 2015 or older (and probably never updated it since 2012 – if they even used the most current one back then). You can make your client talk to it anyway, but for me there’s generally no need for it.

Remote MGMT choices

Definitely go to Maintenance -> Firmware Upgrade and do it now if you haven’t done so already.

Firmware Upgrade

Finally there’s Maintenance -> Configuration. Here you can backup the configuration settings you just made and download an archive to your computer. Doesn’t hurt to do that.

Configuration

So much for the modem. There’s more things it can do but they are mostly only relevant in router mode (and sometimes even then only when you have special requirements).

OPNsense dialup

With the modem fully configured and working, it’s time to configure OPNsense to do the DSL dialup. I chose to rename the first network interface from WAN to PPPoE, but that’s only a name. You need to go to Interfaces -> PPPoE (or whatever yours is called) and change the IPv4 Configuration type from DHCP to PPPoE (unless you have an IPv6-only line of course in which case you’d configure that instead).

OPNsense WAN connection configuration

Further down enter the username and password for the PPPoE connection. Check the documents you got from your ISP, they should be on there somewhere. If they aren’t, ask for them.

OPNsense PPPoE configuration

And that’s all. Save your changes and if everything is correct, OPNsense will do the dialup and establish an Internet connection! Much better now that a trusted device does this, isn’t it?

Conclusion

My new setup is not perfect. Ideally I’d make my OPNsense machine deal with the IP telephony, too. Before even attempting that I will however need to do a lot of reading upfront. So there’s another long-term goal.

Nevertheless this was a change for the better. I made another step in reclaiming my own network. So far I’ve been running this setup for a month and did not face any problems. There has been a short power outage once: After power was back, the APU and the modem booted and before long OPNsense had re-established the connection and I was online again.

What’s next?

The next article will be about building custom packages on OPNsense (since it’s a somewhat involved topic it will probably be split into two posts, though).

FreeBSD router take 2 (pt. 3): Excursion – De-hardening OPNsense for 2022?

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/bsd_router_take_2_pt3.gmi

After OPNsense announced that they would rebase on vanilla FreeBSD instead of going on with HardenedBSD, I wrote the previous article on what “security” means when it comes to networked devices that are connected to the Internet. It also took a look at the fields where FreeBSD is doing pretty good. There’s also the other side of the coin however. Being a person who really likes FreeBSD and enjoys working with it, this article is not one that I looked forward to writing. But FreeBSD is not all sunshine and roses (who would have thought?). And people should be aware of that to make an educated choice. So here we go.

Defaults / (lack of) exploit mitigations

It’s the last two points from the previous article’s list that FreeBSD admittedly sucks at. As discussed there, you can make your FreeBSD systems a lot more secure than they are after a default installation. But FreeBSD does not believe in a “one size fits all” security concept. Truth be told, I’d also be very skeptical about such a concept. Either it will be so general that it’s not worth even laughing about or it will be highly theoretical and probably collide with real deployments pretty soon.

FreeBSD provides a lot of documentation to help you understand the system. However only you know your specific use case and therefore only you can put together the best possible security concept. I strongly support this way of thinking. It is true however that FreeBSD could do a lot better with regards to basic security. Why isn’t a firewall active by default? If it was that would probably be ipfw with a simplistic rule set. One of the first things that I’d do on a new installation would be disabling it and configuring pf instead. And if somebody really has a use case where there should be no firewall – well, nobody would stop you from just disabling it. Coming with any firewall enabled by default would neither limit your choices nor would it make FreeBSD unfit for any sensible scenario. Making a change like that has POLA implications, though (FreeBSD’s “Policy of least astonishment”). But if the will was there, a way would be found.

There’s other things that make it really hard to come up with an excuse for. Frankly speaking: When it comes to mitigation techniques, FreeBSD is hardly a modern operating system. Linux, Windows – basically every common system you can name did a whole lot of work in that area over the last two decades. In comparison FreeBSD did almost nothing. Unfortunately there is not much more to say about that.

And one of the few things that were done, wasn’t done right according to some security researchers. FreeBSD claims to support ASLR (Address space layout randomization). It’s not enabled by default, but it’s there:

# sysctl -a | grep aslr
kern.elf32.aslr.stack_gap: 3
kern.elf32.aslr.honor_sbrk: 1
kern.elf32.aslr.pie_enable: 0
kern.elf32.aslr.enable: 0
kern.elf64.aslr.stack_gap: 3
kern.elf64.aslr.honor_sbrk: 1
kern.elf64.aslr.pie_enable: 0
kern.elf64.aslr.enable: 0
vm.aslr_restarts: 0

The idea of this mitigation technique is to randomly arrange data of processes in memory to make it harder for an attacker to hit a targeted function. In general ASLR’s effectiveness as a mitigation has been doubted by a lot of people who are into security. Then again, it does not come with a high cost and so it’s often seen as a baseline of protection. It’s been standard in OpenBSD since 2003, in Linux since 2005 and in OS X as well as in Windows since 2007. NetBSD was a bit late to the party and only implemented it in 2009. Heck, even Oracle Solaris adopted ASLR in 2012. And FreeBSD? Totally took their time. Got it first in 12.1 which was released in late 2019. But better late than never and maybe the wait was worth it if we got a superior implementation for that?

There’s one problem, though: FreeBSD’s implementation has been criticized as half-baked… According to Shawn Webb it’s not even ASLR but ASR. So we got what we got extremely late, it’s disabled by default and even if you enable it it’s a pretty weak form of what is considered a very basic mitigation. This does not make FreeBSD look too good.

Is FreeBSD well-maintained?

Let’s poke another pain-point, shall we? FreeBSD is much, much less active in cleaning up their system than OpenBSD for example. If you take a look at the source repository, you won’t have to search too long to find things that are not all that pretty. Here’s one example of a commit updating the comcontrol manpage.

The commit removes a reference to the sio(4) interface in the comcontrol(8) manpage. This change is available in the recently released FreeBSD 13.0, older supported releases 12.2 and 11.4 still mention the interface. The thing is that sio(4) was removed from the GENERIC kernel in 2008 – which means that the manpage change that finally shipped this year (and for the very latest version only!) could easily have been shipped with FreeBSD 8.0 over ten years ago…

Want one more? Have a look here. Meet “pnpinfo” which according to the manpage “reports information about Plug-n-Play ISA devices“! Hasn’t been touched in over ten years and is very obviously completely obsolete. It’s not built by default anymore, can’t be built manually either (due to a missing system header file) – but it’s still there in the source tree. It looks like it was still part of FreeBSD 10.0 (early 2014) but removed for 10.1 (late 2014). Till the end, the pnpinfo(8) manpage referenced pnp(4) which in turn had already been removed in FreeBSD 4.6 (2002)!

Right, this is nitpicking around very minor issues. Basically every project has dusty corners and when it is the size of FreeBSD, it would be close to a miracle to not have any. Still it’s only two easy to find examples out of many that show one thing: There’s room for improvement. Plenty actually. But while *BSD prides itself in good documentation, little leftovers like this don’t have such a huge impact on security. Could really be much worse right? For example if little to no maintenance was being done on very important system components responsible for, say secure authentication?

I can hear anything from deep sighs to screams of agony from readers familiar with FreeBSD even before I put a link to Heimdal in the base system here…

Really – FreeBSD ships with Heimdal 1.5.2 in the base system… This version was released in 2012 (!!) and nobody should have trouble believing me that there’s a bunch of nasty CVEs for it. Right, everybody knows that you should never use kerberos from the base system. If you need it, always install either security/heimdal or security/krb5-$version from ports or packages. That way you’ll get versions that are up to date. But honestly: Why the heck is that ancient base system version even there? Nobody should have used it in almost a decade! What’s the point in having a trap like that lurking in base? To see if unsuspecting users might fall into it and make an acquaintance of the poisoned spears at the bottom? That’ll teach them a valuable lesson, eh? No, sorry. No point in even trying to whitewash this. It is just hideous and a real disgrace.

And then there’s of course the recent turmoil around the flawed Wireguard implementation that almost made it into FreeBSD 13. If you don’t know what I’m talking about, consider skimming over this Ars Technica article.

It is actually not a good article and I expected more of Jim Salter; he does a podcast called “2.5 admins” together with FreeBSD developer Allan Jude and they discussed the topic a couple of days before Salter wrote the linked article, forgetting some of the important things and concentrating on minor matters to have “a good story”… It will introduce you to the drama, however. Keep in mind that Jim pretends that the flawed code was “probably” only removed because the original Wireguard inventor intervened even though FreeBSD developers were looking at the code and there already were people unhappy with it (whereas he denies that the even more recent happenings around Linux and the University of Minnesota showed that things in Linux world are also far from perfect).

Bottom line: There’s all kinds of problems in FreeBSD. From small cosmetics to heavy-duty stuff. But FreeBSD is an Open Source project. If you think about contributing fixes (even for the very simple things): By all means do so! It’s not that FreeBSD wants to be in the sorry state it is in regarding certain areas. The project is taking new contributions with open arms. You’d help make the world a little bit better for many people. And there’s plenty of valuable skills to acquire if you choose to go down that road. Doc committers in FreeBSD are equal in their rights to ports and source committers, by the way. If you’ve got a bit of time for it and an interest in tech (you’re reading articles like this not because you don’t care at all, do you?) seriously consider it.

HardenedBSD

At this point the sunshine that the previous article may have shown is probably gone and there are some pretty dark clouds in the sky. Don’t let the problems that I pointed out here scare you away. Remember that the above was written by a FreeBSD user – not a former user. Everything added up, FreeBSD is a decent platform that’s not worse than any other. In fact it has a lot of advantages that help accepting some of the disadvantages. Be aware of the ugly part, though. It might bite you otherwise.

But is this a god-given situation that we cannot do anything about? Is it either the really nice features and sane structure of FreeBSD or better mitigations but much less overall usefulness of OpenBSD (alternatively the better mitigations but the chaotic mess that is Linux these days)? Fortunately not: Enter HardenedBSD.

Have a look at this image to get an idea of what HardenedBSD is doing:

HardenedBSD feature comparison

It’s only four security features listed there that OpenBSD has but HardenedBSD doesn’t. Of course the comparison is not complete, missing out a several good things in OpenBSD like e.g. pledge. However HardenedBSD also has a lot that go even further than what OpenBSD does.

And that’s really, really impressive. Keep in mind that HardenedBSD is basically FreeBSD with a ton of security improvements to it: It has ZFS, jails and all the good stuff. It’s a bit less convenient to use (e.g. you will have to understand additional tools like secadm to toggle certain mitigation features on or off for specific programs). It offers you the means to make system administration a fair bit more cumbersome – while making life terribly hard for attackers. If you are serious about security and accept that there is no free lunch, you’re willing to endure the additional restrictions for a huge gain in hardening your system.

HardenedBSD is a hardened but not a hard fork of FreeBSD. It tracks upstream FreeBSD and merges new code from there. The project also aims to develop security features outside of FreeBSD but to ultimately give the changes back. This would be a huge gain for security-focused FreeBSD users. A very small project however has also very little chance of getting FreeBSD to accept proposed hardening techniques. For that reason HardenedBSD needs every bit of support it can get.

For some time, HardenedBSD also had LibreSSL in base instead of OpenSSL. They had to switch back for the simple reason that the team was to small to keep up with the work required for such an invasive change along with all the other security improvements. And now that OPNsense has announced to ditch HardenedBSD, it will lose some more badly needed support.

So is it a hopeless case? Well, not quite. OPNsense was definitely the most prominent user of HardenedBSD but certainly not the only one. There are people and companies using it. There is being research done with it (see the e.g. this bunker jails article).

Co-founder Shawn Webb also managed to get a foundation started for it and even to attract an impressive amount of donations last year. I’d say that $13,000 instead of 11,000 they had aimed for is not bad at all! Especially if you compare it to the NetBSD foundation which only managed to get about 24,000 of their 50,000 goal even though they are a much older and bigger project.

I’ve been thinking about using HardenedBSD instead of FreeBSD when I build my next workstation. I’ll probably also use it when I reinstall my server and see how that goes. Both will probably things to write about here on the Neunix Gemlog.

Does leaving HardenedBSD make (OPN)sense?

Decisions like this are always a tradeoff and I’m not under the impression that the OPNsense team made this one without carefully considering the matter. In short-term I think that tracking mainstream FreeBSD will definitely benefit OPNsense. Here’s a couple of reasons:

  • It makes development easier in general
  • It will speed up adoption of newer releases
  • It will free resources (e.g. currently the team has to backport fixes to a no longer supported FreeBSD release)
  • It makes debugging easier
  • It might attract additional contributors familiar with FreeBSD but not HardenedBSD

Sounds good, right? If you’re willing to sacrifice the additional hardening of HardenedBSD it sure does. And I think that most people would in fact prefer to go down that route.

IMHO OPNsense is hurting itself in the long run, though. The major reason for ditching HardenedBSD is that it is too much of a niche platform after all. With OPNsense leaving it, it will become even more niche. It is a very important project to eventually take FreeBSD into the right direction. Let’s not underestimate the gem that we have here! Trying to increase adoption would be what we should be doing, not decreasing it further.

But I don’t want to challenge the decision that has been made, write a petition and bring unrest to the community. What OPNsense needs is to continue evolving for the better. One goal that aligns perfectly with the new strategy is getting rid of some more quirks that OPNsense inherited from pfSense and rather doing things like FreeBSD does. This would benefit everybody.

And who knows: Perhaps we’ll see something like “HardenedSense” in the future? Not as a fork but as a community build for people who prefer to stick with a hardened system for their packet filter needs. I hope that this is food for thought for some readers. Maybe we can start a discussion over at the forums or so. If there’s anybody interested in this, please let me know.

Why not OpenBSD?

Following the announcement of OPNsense to part ways with HardenedBSD, some users over on Reddit proposed to rebase on OpenBSD instead. Let’s consider this for a moment.

OpenBSD is generally regarded as a very, very secure operating system. It has a great lot of mitigations in place, a nice and clean codebase and a reasonable-sized community. That’s certainly appealing. People also frequently mention that it has a much newer version of Pf which would be very much beneficial for a project like OPNsense.

There’s a couple of reasons why this is not as good an idea as it seems, though. I actually like what the OpenBSD people are doing. No, truth be told, I admire their security first stance and the fact that they are willing to take it to the extreme anytime. But… Exactly this makes it the wrong choice for anything like OPNsense:

  • Performance is not a primary goal for OpenBSD. If you want a top-notch router, you’ve ruled it out.
  • OpenBSD is a research OS. You can use it in production but you have to live with things like NO ABI stability whatsoever.
  • There’s a lot less software packaged for OpenBSD.
  • OpenBSD does not provide safe data storage, they don’t have any next-gen filesystem.

Let’s also address the misconception of “newer Pf on OpenBSD”: This is not true. Pf originated in OpenBSD when they dropped (due to licensing issues) IPF which they used before and replaced it with their own packet filter. Pf was later ported to FreeBSD (and NetBSD). After those ports happened, OpenBSD continued to improve Pf. One thing that they did was revising the syntax. FreeBSD did not sync their Pf with OpenBSD anymore – but for a good reason! They had improved their version of Pf to make it perform much better with multi-core CPUs. Contributing those changes back to OpenBSD was hopeless since OpenBSD was largely not SMP-capable at that time. For that reason Pf on OpenBSD and Pf on FreeBSD diverged, up to the point where merging newer changes from OpenBSD was simply not feasible anymore.

So it’s not that OpenBSD has “newer PF” – it’s more like both OpenBSD and FreeBSD have distinct versions of Pf that are actively developed but are quite different despite the common name. Rebasing OPNsense on OpenBSD would not give the users a much better Pf. In fact the major user-visible advantage of OpenBSD’s version of Pf – i.e. the simpler syntax – would not even be user-visible on OPNsense as people use the GUI to create their rules! It would on the contrary mean that code changes would be required so that the OPNsense application responsible for the rules would generate the rules in the new syntax expected by OpenBSD’s Pf.

There would also be a lot of other things to change. OpenBSD’s networking works quite a bit differently (e.g. the system’s hostname goes into /etc/myname instead of into /etc/rc.conf as used in FreeBSD). The init system is slightly different. Packaging works very differently (not using /usr/local for example and the package managers are simply worlds apart). And so on.

Conclusion

FreeBSD is a solid operating system that’s doing well overall but is severely lacking in certain areas. HardenedBSD offers all the benefits of FreeBSD without a lot of the weaknesses and is an innovating force when it comes to strong security. OPNsense leaving HardenedBSD behind is a sensible choice considering OPNsense alone but a very unfortunate move for the FreeBSD ecosystem as a whole. OpenBSD is not the right base for OPNsense either.

If you care for FreeBSD and security, please support HardenedBSD. Let’s keep it going strong – maybe there’s the chance of having a community edition of 22.1 and onward that’s still going to be based on HardenedBSD if there is enough interest.

FreeBSD router take 2 (pt. 2): Excursion – FreeBSD and security

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/bsd_router_take_2_pt2.gmi

After I completed the previous article, Franco Fichtner announced that OPNsense and HardenedBSD will be parting ways.

I’m happy to see that they are parting ways in good terms. So there at least were no ugly things going on behind the curtain. The explanations given in the announcement are interesting and I’d say that they make sense. This is a pretty massive change, though. And since (thanks to the friendly Web UI) OPNsense is used by a lot of people who do not have a FreeBSD background, I’d like to explain in a bit more detail what the actual situation is like.

In the first series of posts, the second one was an excursion on using the serial console. This time we’re going to take a look at the broad topic of security.

What is “security”?

FreeBSD has had incredibly talented security officers like Colin Percival, founder of the Tarsnap backup company. His company’s motto is Online backups for the truly paranoid – and it lives up to that. Thanks to him and many great people in the security team, FreeBSD has built up a fairly good reputation regarding security with a lot of people.

There are other voices, too. For example one former FreeBSD user who switched to OpenBSD is industriously working on making FreeBSD look bad. Here’s the homepage where he keeps track of all the things he thinks FreeBSD does wrong.

So which claim is true then? Is FreeBSD doing pretty well or is it downright horrible?

Both of them. And neither. Oh well… It’s a bit too complicated to give a plain and simple answer. So let’s think about what “security” actually means for a moment before returning to judge FreeBSD’s performance in that area.

There are multiple aspects to security. To take the whole situation into consideration means to admit that we’re in one giant mess right now!

Living in a nightmare

We absolutely depend on today’s technology. Think about replacing “the Internet” for example. Even if you have this exceptionally great idea and can provide a concept that is totally sound – how do you think we could get there? Millions of enterprises require the Internet as we know it to continue working as it does. The chances of succeeding with establishing something better that would run in parallel? Basically nonexistent. Nobody could pay for such a huge project! And even if it were to magically appear and be available tomorrow (somehow production-ready by day one), how do you think getting a critical mass of businesses to adapt it?

Incrementally improving what we have is hard enough. If you disagree just think about disabling anything but TLS 1.3 on your employer’s webservers. You, dear reader, probably are ready for such a change, I wouldn’t doubt that. But are all your customers? And that’s only one example of… many.

While being condemned to never being able to “re-invent the wheel” in large scale is unfortunate, it’s not catastrophic. What is catastrophic however is that the very foundations of the technology we’re using today were very much over-credulous from today’s point of view. It’s perfectly reasonable not designing network protocols for security when you don’t think of potential offenders because your network is either limited to one institution or basically to a couple of universities! We’ve outgrown those innocent times for a long time now however. The Internet is a war zone.

If you feel brave, join us and participate in our Gemini experiment (see top of this article). Get a Gemini client, read this article over that protocol. Ideally get your own gemlog started and share original content with the world. While we’re not even dreaming of replacing the Internet with something better – even the very act of challenging the Web alone by providing an alternative for like-minded people who loathe superfluous complexity, is an Herculean effort in and of itself.

But back on topic. Retrofitting security into existing technology that’s already in production use is incredibly hard. Especially if you are supposed to NOT break the former! And if it wasn’t hard enough, this scenario also comes with the curse of optional security which is another pretty sad story by itself… Your DNS server probably supports DNSSEC by now. But does it use DANE (mine doesn’t, yet…)? And how many of the more popular nameservers on the Internet do? I mean, it’s been almost a decade since it was introduced. We need regular “DNS flag days” to force people adapting somewhat acceptable DNS standards. How much can we expect optional security features to come to wide-spread use?

And if all of that wasn’t bad enough, in 2018 we learned that the one most important CPU architecture (x86) has a flawed design that dates two decades back… If you want to refresh your knowledge of what Meltdown and Spectre are, here’s an excellent read (explained so that each and every layman can understand it) by the aforementioned Colin Percival: Some thoughts on spectre and meltdown.

Another point is that while almost everybody tends to agree that security is “important”, few want to actually spend money or effort to improve it. Reading last year’s FOSS Contributor Survey of the Linux Foundation gives you an idea of how bad the state of affairs really is.

It’s 2.3 percent (on average!) of a developer’s time that is spent on work to fix security-related issues in their projects. If you assume a paid developer working 9-to-5, that’s about 11 minutes per workday, adding up to not even an hour a week… It’s a clear trend to rather work on exciting new features than fixing bugs in existing code. Working on security-related bugs is even less popular. How many people with such a mindset would you expect to proactively audit their code for flaws regarding security?

Security as “surviving in a deadly environment”

We set sail and started exploring silent waters around the coast with a pretty much experimental boat. It has been a very exciting ride – because at some point we realized that we were no longer near the coast but somewhere in the middle of the ocean. So what was really a nice toy before has turned into a necessity for us to survive. Quite a while ago we noticed that we were in stormy water and that we had to constantly fix our humble boat when the force of another tide tried to smash it! We’re constantly fixing new leaks and fighting off dangers that nobody really anticipated. And to make it even worse, while a lot of people were interested in tuning our boat for performance, they didn’t want to see that the water around us started boiling. Today we’re surrounded by lava. There is no lifeboat. If you shipwreck, you’re dead.

That picture should have either reminded you of what you already knew – or have been eye-opening regarding our situation. There is no need to panic (that wouldn’t be helpful), but don’t fall into the trap of simply dismissing the shadowy dangers. They are very much real! So whatever helps your employer survive in this situation could be thought of as a security feature.

There is no single panacea but combining a lot of security features can drastically lower the threats. Here are some important bits:

  • Respond to discovered vulnerabilities in a timely manner
  • Offer the latest versions of software or backport fixes
  • Provide means that harden the system
  • Develop mitigations for when (not if!) your first line of defense is breached
  • Make it as easy as possible for the user to secure the system

Where FreeBSD does mostly well

When it comes to patching base system vulnerabilities, FreeBSD is generally doing well (there’s no point in denying that things do not always work like they should so that blakkheim can point a finger at it). If you’ve never read a security advisory as published by the FreeBSD project, I suggest you do so at least once to get an idea. Like the latest one.

As you can see, the security team does not only silently fix bugs but even goes an extra mile, doing a write-up for the interested reader. For years I’ve been very happy with those; I’m not a developer with sharp C skills and deep knowledge of exactly how programs work, but I can always understand what’s going on there. I’m not spiteful enough to ask you to compare them with OpenBSD’s errata which are… very bare-bones.

If you take a closer look at the example provided here, you can see that among the versions that received a fix is 13.0-RC5-p1. That’s right: This means that they even cared enough to fix it for a Release Candidate that has a lifecycle of two weeks! And not only that, they even provided binary updates for that even though people on RC5 could just be expected to update to 13.0-RELEASE only a couple of days later. I’d say that this is nothing short of very commendable acting.

Regarding packages that are not vulnerable, the situation with FreeBSD is a mixed bag. There are quite a few unmaintained ports stuck at older, insecure versions. Common software is usually pretty recent. To give you an idea (and so that you don’t simply have to take my word for it), let’s compare FreeBSD and Ubuntu 21.04. FreeBSD currently has a port count of slightly above 31,000 whereas Ubuntu offers just short of 33,000 packages. A little over 6,000 are outdated on FreeBSD, for Ubuntu it’s over 8,500. For ports beginning with the letter “A”, FreeBSD has 9 ports with versions that contain a known vulnerability (with a CVE) that could be fixed by updating to a newer version whereas Ubuntu has only 3. Same thing for ports that begin with “Z”: 1 vulnerable port that could be fixed by updating to a newer version on FreeBSD, 2 such packages on Ubuntu.

Just so nobody claims I’d selectively present data to support either story with it, here’s a table for package CVEs of all starting letters:

Starting letter FreeBSD # Ubuntu 21.04 #
A 9 3
B 3 3
C 4 7
D 2 4
E 3 0
F 4 1
G 9 5
H 2 2
I 3 4
J 8 3
K 0 1
L 12 13
M 8 3
N 2 18
O 4 6
P 18 17
Q 0 0
R 8 13
S 8 10
T 5 7
U 2 2
V 2 2
W 3 2
X 3 5
Y 1 0
Z 1 2
TOTAL 125 133

I think it’s safe to say: FreeBSD does pretty good in this field, too! Especially if you take into consideration that most of FreeBSD’s ports are done entirely by volunteers and that while software usually “just works” on Linux there’s often some more work required to make it work on other operating systems! (Of course I’m aware that I’m just scratching the surface here and a deeper analysis would be nice – but that would definitely take its own article.)

Let’s talk about means of hardening the system. There’s a lot you can do to harden a system that was installed using the default options. For a while now (I think starting with 11.0) FreeBSD offers a hardening dialog in the installer, allowing for really simple improvement of the defaults. This is one thing that blakkheim prefers to ignore: Yes, /tmp is not cleared by default, but FreeBSD can do that if you want it to and it’s not hard to make it do that.

Yes, hardening a FreeBSD system is not something you can expect the junior admin to master in a couple of hours. But that doesn’t mean that it’s impossible to do. With securelevels and file flags, FreeBSD gives you a powerful tool for increased security. Capsicum and casper are two more things you can take a look at and start making use of. Taking advantage of jailing applications is another great way to make your infrastructure more secure by confining possible intruders and further limiting the damage they can do. FreeBSD expects you to do all that and more, depending on what your security requirements are.

Definitely have a look at security(7). To quote from it:

A little paranoia never hurts. As a rule, a sysadmin can add any number of security features as long as they do not affect convenience, and can add security features that do affect convenience with some added thought. Even more importantly, a security administrator should mix it up a bit — if you use recommendations such as those given by this manual page verbatim, you give away your methodologies to the prospective attacker who also has access to this manual page.

Does that really sound like it’s written by people who do not care for security at all as our friend blakkheim wants you to believe?

So far for the good part. The ugly side of FreeBSD will be covered next time as this article is already way too long. Thanks for reading!

What’s next?

The next post will briefly discuss FreeBSD’s security weaknesses and how HardenedBSD fits into the picture. I’ll also address the “rebase it on OpenBSD!” suggestion some people have made.

FreeBSD router take 2 (pt. 1): OPNsense ZFS-based installation (by converting FreeBSD)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/bsd_router_take_2_pt1.gmi

In 2017 I wrote my longest (by far) series of posts on a single topic: 8 posts on hardware and the FreeBSD-based firewall solutions pfSense and OPNsense. Even after almost four years, some of these posts are still very high on the list of frequently visited pages. A lot has happened since then, though. About time that I get back to the topic! Here’s the list of the old posts:

Part 1 (discussing why you want to build your own router and how to assemble the APU2),
Part 2 (some Unix history explanation of what a serial console is),
Part 3 (demonstrating serial access to the APU and covering firmware update),
Part 4 (installing pfSense),
Part 5 (installing OPNsense instead)
Part 6 (Comparison of pfSense and OPNsense)
Part 7 (Advanced installation of OPNsense)
Part 8 (ZFS and jails)

I meant to revisit this topic again much sooner, but it didn’t work out. In 2018 I started to write a post that (among other text) contained the following paragraphs:

A lot has happened in the meantime. The new pfSense version that I mentioned before has left beta and is available regularly now. And with it comes one real boon: Complete ZFS support! OPNsense had two releases since 17.1 (the version that I discussed): 17.7 and 18.1. I had updated my box to 17.7 without any problems IIRC. After moving houses (and finally being online again after months!) I brushed the dust off my router and connected it. I decided to do a re-install this time and see if anything noteworthy had changed.

First I was a little disappointed to see that the USB3 issue has not yet been taken care off and still makes installing on the APU2 harder than it needs to be. But well, I had figured out how to do it previously, and still had my gear at hand to install using the internal USB2 connection. Second letdown: Still no trace of ZFS to be found. Looks like my priorities do not completely match the ones of the OPNsense team! 😉

But then the pleasant surprise: The 18.1.8 update brought experimental ZFS support! I had done a simple install this time as I had been a bit in a hurry, so I could not test it myself so far. However if I got things right, this means that the boot scripts are finally ZFS-capable and this will likely be supported in the 18.7 release. Chances are that there will not be installer support for ZFS, though. But we’ll see! Good things are going on and eventually we’ll get there.

I didn’t finish the article for reasons that escape me today, but when I wanted to return to the topic another year later, my hardware broke and so I had to postpone that again. My APU has since been fixed and even the old 16GB mSATA drive it originally came with replaced with a nice new one that has much more useful 512 GB of space – but I never found the time to even boot it up once since early 2020.

In late February 2021 I finally got around to install OPNsense on it again and thus get my own router back in business. I took notes but publishing this article was delayed for various reasons. Anyway, here we go! Let’s do a fully ZFS-based installation (which was not possible, yet in 2017).

Back in OPNsense land

When I first built my router, I had to be a bit creative to end up with an OPNsense installation that offered ZFS so that I could use a jails manager on it that depends on that filesystem. However it was only an additional pool for data. The actual system ran on UFS.

For a while now OPNsense does support booting from ZFS! I definitely want that, so let’s give it a try. I download a DVD image for OPNsense 21.1 and put that on my CD emulator device. Then I wire my APU up to my workstation via a serial cable and connect:

# cu -l /dev/cuaU0 -s 115200

Then I boot the machine up. As I’m using the DVD image, it’s configured for VGA mode. That means in my case I have to manually configure it for serial usage. This can be done conveniently in the loader in FreeBSD 12.2 and newer. However… OPNsense 21.1 is still based on version 12.1 where that option is not available, yet. Even though this is an older and in fact unsupported version, I cannot blame the project. They don’t use vanilla FreeBSD directly but are based on HardenedBSD instead, a close fork that is about hardening the code base.

While HardenedBSD has attracted enough attention and people to successfully form their own foundation, it’s still a small project with limited resources. And 12.1 is the only release from FreeBSD major version 12 that they support. If you like to support Open Source projects and have some spare money, consider donating. They are doing important work to move FreeBSD in the right direction!

So we have to do this the old way. Pressing ESC drops me at the loader prompt. Time to set some values and then boot:

set boot_serial=YES
set comconsole_speed=115200
set console=comconsole
boot

Alright! Kernel is booting up and printing all messages to the serial console. It shouldn’t take too long before… Whoo! What’s this?

Mounting from cd9660:/dev/iso9660/12_1_RELEASE_AMD64_CD failed with error 19.

Good lord, I remember this! OPNsense 17.1 (or was it 16.7?) had a problem booting up on some USB3 ports. You had to use a workaround to be able to boot up to the installer… I totally expected that this would have been fixed by now. It’s been four years! *sigh*

I quickly tried out installing from a usb memory stick – but ran into the same problem. So what now? Back in the day I opened up the case and connected an adapter to be able to use an internal USB2 connector. Of course I could do that again. Except – I have no idea whatsoever where I put that before moving houses… A little research doesn’t exactly leave me in a much better mood: As of OPNsense 21.1 they’ve apparently not added an easy way to install to ZFS, yet. At this point I’m ready to question if this project was a good idea or if I should simply do something else.

But this is not as bad as you might think. Fortunately there’s a very simple way of achieving it anyway: Just install vanilla FreeBSD and then use a conversion script. As I had just completed a series on PXE-booting (and the PXE server still at hand), I chose to go down that route. Take a look at that article (and the previous two) if you’re interested in the PXE booting details or don’t understand something that I do in the next section.

On the very day I’m finally writing this article though, the OPNsense team has announced switching the installer for the upcoming version 21.7, making installation to ZFS a regular installation option. So that’s one thing to look forward to if you like ZFS but don’t want to use the bootstrap script. 🙂

Installing FreeBSD

So I boot up my PXE server. It was prepared to serve FreeBSD 12.2. It is a very bad idea to try to bootstrap OPNsense on a version of FreeBSD that’s newer than the one the firewall OS is based on. So for this special case I add FreeBSD 12.1 support on my PXE server:

# mkdir /usr/local/www/pxe/bsd/fbsd/amd64/12.1-RELEASE
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.1-RELEASE/MANIFEST -o /usr/local/www/pxe/bsd/fbsd/amd64/12.1-RELEASE/MANIFEST
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.1-RELEASE/base.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.1-RELEASE/base.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.1-RELEASE/kernel.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.1-RELEASE/kernel.txz

By default, PXE-booting is disabled on the APU. So when turning on the device without any USB device attached to it, this is what I get on the serial console:

Press F10 key now for boot menu

Select boot device:

1. Payload [setup]
2. Payload [memtest]

So let’s see what options the setup has to offer:

Boot order - type letter to move device to top.

  d SATA
  c mSATA
  b SDCARD
  a USB
  e mPCIe1 SATA1 and SATA2
  f iPXE (disabled)


  r Restore boot order defaults
  n Network/PXE boot - Currently Disabled
  u USB boot - Currently Enabled
  t Serial console - Currently Enabled
  o UART C - Currently Enabled
  p UART D - Currently Enabled
  m Force mPCIe2 slot CLK (GPP3 PCIe) - Currently Disabled
  h EHCI0 controller - Currently Disabled
  l Core Performance Boost - Currently Enabled
  i Watchdog - Currently Disabled
  j SD 3.0 mode - Currently Disabled
  v IOMMU - Currently Disabled
  y PCIe power management features - Currently Disabled
  w Enable BIOS write protect - Currently Disabled
  x Exit setup without save
  s Save configuration and exit

Ok, using “n” I can enable PXE booting. Next time I get this message:

Press F10 key now for boot menu, N for PXE boot

Pressing N takes me to the iPXE boot menu. I quickly interfere to manually specify what to do instead of letting it automatically guess (and guess wrong). At the iPXE prompt I ask it to get an ip address and then chainload pxelinux like this:

iPXE> dhcp
Configuring (net0 00:0d:b9:42:67:64)...... ok
iPXE> chain pxelinux.0

There we are. As the mfsBSD image that I use is FreeBSD 12.2-based, I can set the console to serial in the loader (by pressing “5”) and then boot. Once the system is up, I login as root with the password mfsroot. Next is preparing the manifest and then starting the familiar FreeBSD installer:

# mkdir -p /usr/freebsd-dist
# fetch http://10.11.12.1/bsd/fbsd/amd64/12.1-RELEASE/MANIFEST -o /usr/freebsd-dist/MANIFEST
# bsdinstall

I want a minimal, bare-bones FreeBSD system and thus choose to not install any optional distsets. When asked about the installation source, I pick “other” and enter http://10.11.12.1/bsd/fbsd/amd64/12.1-RELEASE as that’s where my PXE server offers the required files. Now I can install FreeBSD using root-on-ZFS and everything as I like it.

When the installation is done, I reboot.

Converting FreeBSD to OPNsense

I enter setup again and disable network booting. As this is 12.1 again, I need to manually enable the system for use with the serial console. So when the loader menu comes up, I escape to the loader prompt and again use the same commands as above to set the variables to the right values and then boot.

After logging in as root, I download the conversion script:

# fetch --no-verify-peer https://raw.githubusercontent.com/opnsense/update/master/bootstrap/opnsense-bootstrap.sh

again, we’re using 12.1 which is the last version without without a certificate trust store in the base system. I could install ca_root_nss, but I’m choosing to just not validate the TLS cert this one time (the bootstrap process will do it correctly then).

All that’s left to do now is running the script:

# sh opnsense-bootstrap.sh
This utility will attempt to turn this installation into the latest
OPNsense 21.1 release.  All packages will be deleted, the base
system and kernel will be replaced, and if all went well the system
will automatically reboot.

Proceed with this action? [y/N]: y

It will then install packages and eventually the base system:

Fetching base-21.1.1-amd64.txz: ........................... done
Fetching kernel-21.1.1-amd64.txz: ....... done
!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!
! A critical upgrade is in progress. !
! Please do not turn off the system. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Installing kernel-21.1.1-amd64.txz... done
Installing base-21.1.1-amd64.txz... done
Please reboot.

Then it automatically reboots.

OPNing my senses

Settings for the serial console need to be entered again at the loader once more. Conversion worked like a charm: OPNsense boots up just fine. I’m starting over fresh here, so no configuration importer for me. But yes, I do want manual interface asignment.

For my box I do the following settings: No VLANs, manually configure: igb0 -> WAN, igb1 -> LAN, igb2 -> Opt

Once it’s done, I can connect from my workstation on the LAN to it by opening 192.168.1.1 in a browser. Login credentials are user root & password opnsense (yes, the conversion script wiped whatever root password you chose before).

Then I complete the wizard which mostly means changing the login password. Afterwards I go to System -> Firmware -> Settings and change the Firmware Flavor from the default OpenSSL to LibreSSL (don’t forget to save). Next is checking for updates and updating the system.

The Web UI is nice and all, but I’m a keyboard and terminal person. So for convenience I add a new user for me. To do so, I go to System -> Access -> Users. Obviously a username is required. I don’t want a password for that user, so I leave that blank and check the box “Generate a scrambled password”. My login shell of choice is /bin/tcsh (as zsh is unfortunately not in FreeBSD’s base system). Since I need a privileged user, I add “admins” to the group memberships. Then I paste in my public SSH key (from ~/.ssh/id_ed25519.pub on my workstation – if you’re using a different algorithm use the correct file name for that) and save.

The last thing that I do for settings is allowing my user to actually SSH into the box. So I’m going to System -> Settings -> Administration. There I check the box “Enable Secure Shell” and set “Listen Interfaces” to “LAN”. I’m also allowing passwordless sudo for “wheel,admins”. After I saved the settings, let’s see if I can connect to the box via SSH and become root:

% ssh 192.168.1.1
Enter passphrase for key '/home/kraileth/.ssh/id_ed25519':
----------------------------------------------
|      Hello, this is OPNsense 21.1          |         @@@@@@@@@@@@@@@
|                                            |        @@@@         @@@@
| Website:      https://opnsense.org/        |         @@@\\\   ///@@@
| Handbook:     https://docs.opnsense.org/   |       ))))))))   ((((((((
| Forums:       https://forum.opnsense.org/  |         @@@///   \\\@@@
| Code:         https://github.com/opnsense  |        @@@@         @@@@
| Twitter:      https://twitter.com/opnsense |         @@@@@@@@@@@@@@@
----------------------------------------------

% sudo -i
*** OPNsense.localdomain: OPNsense 21.1.1 (amd64/LibreSSL) ***

[...]

  0) Logout                              7) Ping host
  1) Assign interfaces                   8) Shell
  2) Set interface IP address            9) pfTop
  3) Reset the root password            10) Firewall log
  4) Reset to factory defaults          11) Reload all services
  5) Power off system                   12) Update from console
  6) Reboot system                      13) Restore a backup

Enter an option:

There we go. Now it’s time to configure my firewall settings. As this is a topic of it’s own, I’m going to skip this here.

Conclusion

Just like 4 years ago, there are still some hurdles to overcome to install OPNsense on my router (especially the USB3 issue). There’s also the fact that it’s based on FreeBSD 12.1 instead of 12.2 which is not ideal. But again: I’m not really complaining about that; HardenedBSD chose to concentrate on 13.0 and taking the resources of their small team into account, this was a perfectly reasonable decision. But regarding things like base system certificates and especially console selection in the loader… Well, you adapt to nice new features incredibly fast, making things you had to do before that (and did for years!) a bit of a nuisance. 😉

I started with OPNsense 17.1, I think and did a lot of point release updates as well as a couple of major release upgrades before my hardware went bad during the 18.7 or early 19.1 life cycle, I think. It had never failed me. Being able to use root-on-ZFS today is definitely nice progress. Since I installed the system in early March, a couple of updates came out. It’s great to see how reliably everything still works. So I’d say that I’m back on track – and I’ll definitely have a couple of things that I want to achieve (and write about) this time. Stay tuned!

Building a BSD home router (pt. 8): ZFS and jails

Previous parts of this series:

Part 1 (discussing why you want to build your own router and how to assemble the APU2),
Part 2 (some Unix history explanation of what a serial console is),
Part 3 (demonstrating serial access to the APU and covering firmware update),
Part 4 (installing pfSense),
Part 5 (installing OPNsense instead)
Part 6 (Comparison of pfSense and OPNsense)
Part 7 (Advanced installation of OPNsense)

Fixing swap

This is the last part of this series of building a BSD home router. In the previous article we did an advanced setup of OPNsense that works but is currently wasting valuable disk space. We also configured OPNsense for SSH access. Now let’s SSH in and su – to root and continue! Choose shell (menu point 8) so that we can have a look around.

# df -h
Filesystem           Size    Used   Avail Capacity  Mounted on
/dev/ufs/OPNsense    1.9G    909M    916M    50%    /
devfs                1.0K    1.0K      0B   100%    /dev
/dev/ada0s1b         991M    8.0K    912M     0%    /none
devfs                1.0K    1.0K      0B   100%    /var/dhcpd/dev

Uhm… ada0s1b is mounted on /none? Seriously? Let’s get rid of that real quick:

# umount /none

How did that happen? This leads to the question: What does our disklabel on slice 1 look like?

# gpart show ada0s1
=>      0  6290865  ada0s1  BSD  (3.0G)
        0       16          - free -  (8.0K)
       16  4194288       1  freebsd-ufs  (2.0G)
  4194304  2096561       2  freebsd-ufs  (1.0G)

There you have it. The second one is all wrong, it’s not meant to be UFS! We have to correct it to have proper swap space configured:

# gpart delete -i 2 ada0s1
ada0s1b deleted
# gpart add -t freebsd-swap ada0s1
ada0s1b added
# swapon /dev/ada0s1b
# swapinfo 
Device          1K-blocks     Used    Avail Capacity
/dev/ada0s1b      1048280        0  1048280     0%

That’s better. Now we need to adjust fstab to make this change persistent:

# vi /etc/fstab

Change the ada0s1b line like this:

/dev/ada0s1b		none		swap	sw		0	0

Ok, we have some swap now, but we’re wasting most of the disk space of our drive. Let’s address that one next!

Preparing the system for ZFS

In the installer we created a second slice (MBR partition) as a placeholder:

# gpart show ada0
=>      63  31277169  ada0  MBR  (15G)
        63   6290865     1  freebsd  [active]  (3.0G)
   6290928  24986304     2  !57  (12G)

Let’s delete it and create a second FreeBSD slice instead:

# gpart delete -i 2 ada0
ada0s2 deleted
# gpart add -t freebsd ada0
ada0s2 added

Now we need to create a disklabel inside and create a partition for ZFS:

# gpart create -s bsd ada0s2
ada0s2 created
# gpart add -t freebsd-zfs ada0s2
ada0s2a added

OPNsense does not load the ZFS kernel module by default. So let’s do that now and also notify the loader to always insert that ko during startup (we’re using loader.conf.local because OPNsense overwrites loader.conf during startup):

# kldload zfs
# echo zfs_load=\"YES\" >> /boot/loader.conf.local

Then we set the ashift. This tells ZFS to adjust to a 4k blocksize which is better for most of today’s drives use instead of 512 byte ones, even though a lot of them will lie to you and claim to have 512 byte sector size. But even on a drive that really has 512 byte sectors, using 4k is better than using 512 bytes on a 4k sector drive. You will only lose some space if you have a lot of very small files in this case. In the other case however, you will hurt performance badly. If you know your drive and you want to use another blocksize, look up how to do it. Otherwise just set the ashift like this:

# sysctl vfs.zfs.min_auto_ashift=12
vfs.zfs.min_auto_ashift: 9 -> 12

With that we’re good to go and create a pool and some datasets.

Pool creation

I’m calling my pool zdata but feel free to name yours whatever you like better. I also enable compression on the pool level and turn off atime:

zpool create -O compression=lz4 -O atime=off -O mountpoint=none zdata /dev/ada0s2a

Next is creating some basic datasets that won’t be used directly (hence forbidden to mount) but only serve as parents for other datasets:

# zfs create -o canmount=off -o mountpoint=none zdata/var
# zfs create -o canmount=off -o mountpoint=none zdata/usr

Let’s move the old log dir and create some new directories:

# mv /var/log /var/log.old
# mkdir /var/log
# mkdir /usr/ports

On with some more datasets:

# zfs create -o mountpoint=legacy zdata/var/log
# zfs create -o mountpoint=legacy zdata/usr/ports
# zfs create -o mountpoint=legacy zdata/usr/obj

To make the system use those we need to add them to the fstab:

# vi /etc/fstab

Add these lines to the file:

zdata/var/log		/var/log	zfs	rw		0	0
zdata/usr/ports		/usr/ports	zfs	rw		0	0
zdata/usr/obj		/usr/obj	zfs	rw		0	0

Once these additional lines are in place, the datasets can be mounted and the old logs transferred to their new place:

# mount -a
# mv /var/log.old/* /var/log/

The directory /var/log.old is no longer needed, but the system currently has some file descriptors open that prevent deleting it. Just rmdir after the next reboot. Speaking of which: It is now a good time to do updates (and change the firmware to the libressl-based one if you haven’t switched already).

BTW: Don’t try to put everything on ZFS! I made some experiments booting into single user mode and moving over /usr and /var. The results were… not pleasing. After doing some reading I found that while OPNsense works well with ZFS datasets, it’s startup process doesn’t cope with ZFS very well. Place its configuration on ZFS and you’re left with a partially defunct system (that doesn’t know its hostname and won’t start a lot of things that are needed).

Full ZFS support is already on the wish list for OPNsense. It looks like that won’t make it into 17.7, but I’m pretty sure that it will eventually be available, making root-on-ZFS installations possible. Yes, pfSense already has that feature in their betas for the upcoming version 2.4. And they even ditched the DragonFly installer and use the familiar BSDinstall which is really cool (dear OPNsense devs, please also take this step in the future, it would be greatly appreciated!).

Is this a good reason to switch to pfSense? It might, if for you this is the one killer feature and you are willing to let go of OPNsense’s many improvements. But there’s one big blocker: If you make the switch you don’t really need to read on. You won’t be able to create jails easily. Why? Because pfSense heavily customizes FreeBSD. So heavily in fact that you cannot even use the ports tree by default! And that is truly a rather sad state of affairs. Sure, a lot of pfSense users actually use MacOS or even Windows and only want to ever interact with the GUI. BSD means nothing to them at all. But if you’re a FreeBSD user it’s pretty annoying if things simply don’t work (and OPNsense shows that there’s no real need to screw things up as much as pfSense does it).

Ports and jails

The OPNsense team provides packages for OPNsense that you can simply install via pkg. However they currently offer only 368 packages, so chances are that you want something that is not there. The FreeBSD ports tree on the other hand means that over 27,000 programs are easily available for you! So since OPNsense is based on FreeBSD (and tries to remain close to it) this is really an option.

On FreeBSD you’d probably use portsnap to get a snapshot of the current ports tree. This won’t work in our case since OPNsense doesn’t have that tool. The other common way on FreeBSD is to use svnlite and checkout the ports tree from the Subversion repo. Again OPNsense doesn’t provide that tool. And it also doesn’t package the full SVN.

So what can we do to acquire the ports tree? OPNsense does provide a git package and the FreeBSD project offers a git mirror of the SVN repositories. But wait a second! OPNsense works together with the HardenedBSD team and they have their own ports tree (based on the vanilla FreeBSD one with some additions). The whole ports tree is pretty big, but we don’t really want (or need) the whole history. Just what various version control systems call “head”, “tip”, “leaf”, … For git we can achieve this setting the “depth” to 1:

# pkg install git
# git clone --depth=1 https://github.com/HardenedBSD/hardenedbsd-ports.git /usr/ports

FreeBSD ships with OpenSSL in base and a lot of ports expect to link against that. We’re however using LibreSSL and so we have to tell the build system to use that by making an entry in make.conf:

# echo DEFAULT_VERSIONS+=ssl=libressl >> /etc/make.conf

If – for whatever reason – you decided to stick to the OpenSSL firmware, you still need to edit make.conf. This is because OPNsense uses OpenSSL from ports which is usually newer than the version from base (that cannot be upgraded between releases for ABI stability reasons). Use ssl=openssl in that case.

The next step is optional, but I recommend installing a tool for dealing with ports. My example is a pretty light-weight port but maybe you want to build something more demanding. Especially in those cases a ports management tool comes in very handy. I suggest portmaster which is extremely light-weight itself:

# make -C /usr/ports/ports-mgmt/portmaster install clean

Once you have it installed, you can install the jail management tool. Yes, I know that I’ve written about py3-iocage a while ago, but that comes with a lot of dependencies and doesn’t provide enough of an advantage over the purely shell based iocell fork. For that reason I would simply go with that one in this case:

# portmaster sysutils/iocell

Alright! Now you have iocage installed and can start creating jails. What services would you want to jail on a small router box that is always on? Think about it for a moment. There are many great possibilities (I’ll likely write another article soon about what I have in mind right now).

Looking back – and forward

What have we accomplished in this series? I now have a frugal little router on my desk that is quietly doing its work. So far it’s just an additional machine between my network and the modem/router box from my ISP. Taking a break from topics directly related to the actual router, I’ll setup some jails (and NAT) next. But then there is a lot more to look into: How to do proper firewalling? What about traffic shaping? How to configure logging? Also VPN and VoIP come to mind as well as NTP, a DNS cache or even vLANs or intrusion detection.

OPNsense places so many tools within reach of your hands. You only have to grab one of them at a time and learn to use it. That’s what I intend to do. And then, some point in the future, equipped with much more solid networking knowledge, I’ll try to replace that box I got from my ISP with my own modem, too. But excuse me now, I have some reading to do and configurations to break and fix again.

Building a BSD home router (pt. 7): Advanced OPNsense installation

Previous parts of this series:

Part 1 (discussing why you want to build your own router and how to assemble the APU2),
Part 2 (some Unix history explanation of what a serial console is),
Part 3 (demonstrating serial access to the APU and covering firmware update),
Part 4 (installing pfSense),
Part 5 (installing OPNsense instead) and
Part 6 (Comparison of pfSense and OPNsense)

Revisiting the initial question

In the first post I asked the question “Why would you want to build your own router?” and the answer was “because the stock ones are known to totally suck”. I have since stumbled across this news: Mcafee claims: Every router in the US is compromised. Now Mcafee is a rather flamboyant personality and every is a pretty strong statement. But I’m not such a nit-picker and in general he’s definitely right. If you have a couple of minutes, read the article and/or watch the short Youtube interview that it has embedded.

If you care about things like privacy at all, we’re living in a nightmare already and things keep getting worse. What I have blogged about in this series of posts so far is not really solving any problem. It’s just a first step to take back your network. Have you built your own router, too, or are you planning to do so? Just assembling it and installing a firewall OS on it won’t do the trick. As a next step you have to learn the basics of networking and firewalling so you can configure your box according to your needs. And even then you have just put your own router behind the modem/router box from your ISP and not replaced that. I’d like to go further and get my own modem, too. But that step requires a lot more reading before I will even attempt to do it.

Manual installation

However this article is about doing a more advanced OPNsense installation that leaves room for customizing things. Let’s get to it!

OPNsense Installer: Manual installation

In the installer select “manual installation” obviously. This will lead you through a couple of dialog windows that let you customize your partitioning etc.

OPNsense Installer: Format the disk?

It seems like OPNsense can be installed on an existing filesystem. There might be people who would want that feature but I don’t. I definitely prefer to start fresh as a newly installed OS should be in a clean state in my opinion.

OPNsense installer: Geometry confirmation

The installer then gives you the option to change the disk geometry. You almost certainly don’t want to do this. If you do need to, you have a strange disk, are aware of its quirks and know geometry matters good enough that you definitely don’t need my advice on it.

OPNsense installer: Slice disk?

Next you are asked if you want to slice (OPNsense uses the term “partition” to describe MBR partitions which is fine since that’s what non-BSD people usually call it). I don’t expect to be dual-booting my box or anything, so I could go with just one slice. However I might install and try out some other versions (or take a look at pfSense again when 2.4 is officially out or even something like OpenWRT, just to take a look at it). For that reason I create two slices so I can keep my OS on one and my data on the other.

OPNsense installer: Disk slicing

I created a FreeBSD slice and one of type Plan9. No, I’m not going to put Plan9 on there. It will be erased and re-purposed anyway. But the installer has this option and Plan9 is cool. 3 GB for OPNsense should be enough and I give the rest to the future data slice.

OPNsense installer: Slice alignment

For the advanced installation we’re unfortunately stuck with installing on the MBR partitioning scheme. That means (for compatibility’s sake) the system enforces the old CHS (Cylinder, Head, Sector) addressing limitations which are almost completely irrelevant today, but meh. The most annoying consequence of this is that “partitions have to end on a cylinder boundary”. If you don’t know what that means: It’s related to the physical geometry of spinning drives that has been of high importance in the olde days(tm) and still haunt us today because operating systems are used to work with it (even though geometry parameters have been lies and lies for decades now and SSDs don’t have spinning parts but claim to have them to make the OS happy…). To comply with this, choose to grow or shrink your slice by a couple of sectors.

OPNsense installer: No bootblock installation

If you want to dual-boot (or multi-boot) your box, make sure to install a boot manager now. I don’t anticipate to install more than one OS on it at the same time and so I skip this. Oh, and please don’t ask me what “packet mode” is! I tried to research it, but all that I found boils down to “if you have problems, try with/without it”. I couldn’t really find anything about what that actually does (at least not in a reasonable amount of time). If you know: Please leave me a comment!

OPNsense installer: Slice selection

Next is selecting which slice to install to. Why, the FreeBSD one, of course!

OPNsense installer: Adding disklabel partitions

Finally the slice needs to be partitioned (or sub-partitioned if you regard the slices as “partitions”!). This means that BSD disklabels are created inside the MBR slice to allow for multiple partitions. For the setup that I have in mind, two partitions suffice: One for / and the other for SWAP space. For whatever reason the installer does not directly allow to assign SWAP, so I allocate 2 GB for the root partition and the rest to a second partition that has no mountpoint. That’s it, the installation can start now.

SSH access

Once the installation is complete, follow the steps that I wrote about in the article about the simple installation.

Got the interfaces assigned and the setup wizard run? Good. OPNsense can be administered purely through the Web GUI. Howver if you’re like me, you really prefer some means of direct console access. Sure, we have that over the serial console. While that’s fine for the installation, it’s a bit cumbersome for daily use. Fortunately there’s a better way: Let’s just enable SSH access!

OPNsense Web GUI: Creating a user

First stop: Creating a user (you wouldn’t want to SSH in as root, do you? Do you?! Do this on a production machine like never). One thing is important here: Make your new user part of the “admin” group or else it won’t be terribly useful to you. Also use SSH keys instead of passwords. If you haven’t ever used keys, set a couple of minutes aside to do a little reading about what they are. They are much more secure than passwords and you definitely want to use them even if you don’t know that just yet (I recommend the article on SSH keys over at the Arch Linux wiki. Unless you’re using the original OpenBSD OpenSSH, we’re all using the same version of OpenSSH-portable anyway). You must also check the “use scrambled password” checkbox because OPNsense won’t let you get away with an empty password.

OPNsense Web GUI: Enabling SSH

Then OpenSSH needs to be activated. If – for whatever reason – you cannot use keys, you have to enable the “permit password login” option. Try to avoid that, though. And don’t check the “permit root user login” however convenient it might be!

SSH login to the OPNsense box

That’s it, you can now log into your box using SSH. Use su – and the root PW to become root. OPNsense will then display the nice menu that you already know from connecting via serial.

What’s next?

Right now we have a lot of disk space wasted and there’s other things wrong, too. So after the installation there’s some more work to do, some packages to install, filesystems to create, etc. I originally intended to stuff more into this post but it’s certainly long enough already. See you in part 8, the last part of the series!

Building a BSD home router (pt. 6): pfSense vs. OPNsense

Part 1 of this article series was about why you want to build your own router, and how to assemble the APU2 that I chose as the hardware to build it from. Part 2 gave some Unix history and explained what a serial console is. Part 3 demonstrated serial access to the APU and showed how to update its firmware. Part 4 detailed installing pfSense, while the previous one did the same with OPNsense.

A little overview: In this post I will give you some background information, compare the appearance / usability of both products and then take a look at some special features before giving a conclusion.

pfSense vs. OPNsense: Who wins?

This article is about comparing both products and helping you to make a decision. It is not terribly in-depth, because that task would require its own series of articles (and a lot more free time for me to dig much deeper into the topic). But still there’s a lot you may want to know to get a first impression on which one you should probably choose. If you do some more research and write about it, please let me know and I will happily link to your work!

I want to point out one thing right at the beginning: Both products are good firewall solutions with a heck of a lot of extras. If you have the same goal as I have (building a home router), either will do absolutely fine. That does of course not mean that your choice doesn’t matter at all. You can definitely benefit from thinking about it before making a decision. But even making the “wrong” decision doesn’t mean that it will be horribly wrong. There are a couple of differences and maybe they are important to you. But chances are that both products would completely satisfy your needs.

Heritage

Sometimes it’s helpful to ask the old question: “Where do we come from?” While this question is usually a philosophical one, in our case it helps to shed some light on our topic. If you do a little reading on the net, you will soon find that pfSense and OPNsense do not like each other much. In fact it’s probably safe to state that they are more or less hostile to each other. OPNsense is a fork of pfSense. Obviously not a friendly fork.

Some pfSense enthusiasts have been spreading information about OPNsense which suggests that new team has no idea what they are doing. They are said to frequently break important things and that the whole project is actually quite laughable. Or to put it short: You really should not waste your time with it and stick to the original. Having liked pfSense for years, I would have believed that, even though you should listen to the other side, too, before doing so. But listening to both sides takes time and effort – both of which were rather limited when I briefly looked into the whole clamor in mid 2015.

Eventually it was the plain hatred of one person who appeared to really have no life, that made me look at what the other project would say. This strange guy popped up in every single pfSense vs. OPNsense discussion and threw so much dirt at OPNsense that I could not help but pity that person. The fact that pfSense (despite obviously being completely dependent on pf, a technology that came from OpenBSD) has a rather bad name with a lot of OpenBSD people for behaving poorly in the past, didn’t help to regain my faith in it, either.

According to OPNsense, they were not happy with the code-quality of pfSense. They didn’t like the fact that the whole Web GUI ran as root (ouch!) and wanted to do privilege separation (which is actively work in progress as I was told). Also there were licensing issues when Netgate acquired pfSense and a bunch of other things. Deciso, a company based in the Netherlands, had been a sponsor of pfSense for years but felt that the whole project was going in the wrong direction after Netgate took a couple of actions. So they decided to fund a fork instead.

Who is right? There’s probably some truth to both versions. OPNsense has followed a rapid development style, bringing in lots of new features and even making some rather drastic changes. It’s true that especially in the beginning there were some problems due to that. But it’s also true that they were quick to react to those. One thing that is not true (or at least not the whole truth, if you will) is that pfSense is the original and OPNsense is a cheap rip-off! What’s the whole truth then?

Once upon a time… in 2003 there was a new firewall OS called m0n0wall. Manuel Kasper had built it on a stripped down version of FreeBSD. There had been small firewalls before, but Kasper’s innovation was to put a Web GUI on top of it so that the firewall’s settings could be controlled from the browser! It did not take long and m0n0wall took the world by storm. However Kasper’s project focused on embedded hardware. So only a while later a fork was created which geared towards more powerful hardware. The fork’s name? You’ve guessed it: pfSense. In 2015 Manuel Kasper officially ended the m0n0wall project (because recent versions of FreeBSD had been grown too big to be easily usable for what he did with it in the past). And guess what he did: He gave his official blessing and recommends to migrate to and support OPNsense!

Appearance

Knowing some background is nice, but what do both products feel like? The first major difference between the two is what they look like. This is pfSense’s main dashboard:

The pfSense dashboard

Compare it to OPNsense’s version of the dashboard:

The OPNsense dashboard

As you can see, OPNsense did a lot to provide the user a much more modern GUI. Both dashboards are customizable but it’s hard to argue that OPNsense’s is not superior. But to be fair: pfSense is working on a GUI overhaul as well.

Let’s compare a couple of the menus. This is pfSense’s “System” menu:

pfSense: “System” menu

And here’s the one from OPNsense:


OPNsense: “System” menu

While pfSense uses pull-down menus at the top, OPNsense has a navigation bar to the left. As you can see, both have not that much in common. This is because OPNsense did not only redesign the GUI but also re-arranged which options go where. I find the new arrangement more logical (e.g. with pfSense logout is in “System” but halt is in “Diagnostics”). But that’s definitely a matter of taste.

Here’s what the “Services” menu from pfSense looks like:


pfSense: “Services” menu

And here’s the corresponding one from OPNsense:


OPNsense: “Services” menu

This time there seems to be quite a bit more consensus about what counts as a service. But still OPNsense looks more like a cleaned up version.

Another example is the “Diagnostics” menu in pfSense:


pfSense: “Diagnostics” menu

There’s no direct equivalent with OPNsense. In the “Service” menu above you can see that there is a “Diagnostics” entry. The same goes for “System”, “Interfaces”, “Firewall”, etc.

And now for the heart of the whole thing – here’s pfSense’s (default!) WAN firewall rule settings:

pfSense: WAN firewall rules

Compare that to the same thing from OPNsense:

OPNsense: WAN firewall rules

This is where it shows that both products do have a lot in common: What we can see here is basically the same thing. Again OPNsense simply has the more modern interface.

To end the visual comparison let’s look at the LAN firewall rules from pfSense, too:

pfSense: LAN firewall rules

And here’s the LAN rules from OPNsense:

OPNsense: LAN firewall rules

No surprise here: It’s all very similar just with interface improvements on OPNsense’s side.

Technical

So far it’s mostly a matter of taste. But now on to the technical points. This is where OPNsense shines (which is no wonder since it’s developing a lot faster). For example OPNsense is already based on FreeBSD 11.0 whereas pfSense is 10.3-based. However there’s already beta versions for the upcoming pfSense 2.4 which are also based on 11.0 and feature many more improvements.

One major difference between the two is that pfSense heavily customized FreeBSD while OPNsense believes in the opposite and tries to be as close to mainline FreeBSD, just adding packages on the top of it. I like that approach better but again that’s probably a matter of taste as well.

The most important thing for me is that OPNsense entered into a partnership with the HardenedBSD project. This resulted in OPNsense being able to change the crypto framework used! For me this is the one killer feature. Give me the option to rip out OpenSSL and use LibreSSL instead and I’m sold! However that’s not even all, yet.

OPNsense: Selecting alternative firmware

OPNsense got HardenedBSD’s ASLR (Address Space Layout Randomization) implementation and the most recent addition is the introduction of packages compiled with SafeStack. This is what the most current update notice looks like (it didn’t fit completely on the screen):

OPNsense: Example of an update notice

If you ask me, hardening your router (especially if it should happen to be promoted to be your border router eventually) makes a lot of sense.

There would be much more to write here, but if you’re interested in that, you will probably have to read some of the recent change notes yourself.

Conclusion

OPNsense and pfSense are quite similar in their core functionality. When should you choose which one? Have a look at the pros and cons of each one and decide for yourself!

pfSense:

  • + Better known brand (= more material about it on the net)
  • + 2.4 will feature BSDinstall including root-on-ZFS
  • +/- Slower development
  • Missing a lot of innovations / security enhancements compared to OPNsense
  • Has been acting quite unfriendly in the past
  • Rather intransparent future direction

OPNsense

  • + Already based on FreeBSD 11.0
  • + Nice new GUI, sensible rearrangement of items
  • + Pushing for priv. sep., uses current technology (PHP 7, phalcon 3, …)
  • + Partnership with HardenedBSD (optional use of LibreSSL, SafeStack, …!)
  • + Closer to unaltered FreeBSD
  • + Bootstrap script to turn a vanilla FreeBSD installation into OPNsense
  • + Multiple languages supported
  • + Public roadmaps for future versions
  • + Official blessings from m0n0wall founder Manuel Kasper
  • +/- Fast-paced development, rapid update policy
  • (Currently) less well-known than its competitor
  • (Currently) installer problems with some USB3 devices
  • Not currently able to install root-on-ZFS

What’s next?

The next article of this series will give an example of an advanced install of OPNsense that lets you use the APU for more than just a router!

Building a BSD home router (pt. 5): Installing OPNsense

Part 1 of this article series was about why you want to build your own router, and how to assemble the APU2 that I chose as the hardware to build this on. Part 2 gave some Unix history and explained what a serial console is. Part 3 demonstrated serial access to the APU and showed how to update its firmware. The previous article detailed installing pfSense.

This post will show how to install OPNsense, a great alternative to pfSense.

Preparation

OPNsense was forked from pfSense (more on than in the next post) and thus you will find lots of similarities if you have read the post on installing pfSense. The OPNsense team decided to move forward more quickly and did lots of interesting but invasive changes. One strong point for example is that it is already based on FreeBSD 11.0. There is one drawback to this, however: a problem with the XHCI (USB3) driver can lead to the installation media not being able to mount the filesystem and boot up. This makes installing OPNsense a little bit more complicated since the APU2 only has UBS3 ports.

Well, the board does have an internal USB2 controller, too. Therefore I suggest getting a cable that allows connecting USB devices to it. If this is not for you, take a look at the end of the post, I’ve prepared a section “alternative installation methods” there.

First download an image (select amd64 + serial). Then dd it onto an unused memstick and prepare the serial connection (take a look at the previous posts if you need help with dd’ing or attaching the serial console).

Open APU2 box with serial connection and memstick attached to the internal USB2 controller

As you can see, I’ve attached a memstick with OPNsense via USB2 and made a serial connection. That way the installation works just fine.

Step 1: Installation

Hit F10 to go to the boot menu as soon as SeaBIOS offers it.

Boot menu to select which device to boot off of

Since we’ve attached the memstick over USB2, the internal drive would take precedence over it in the default boot order. So in this case I have to select 2 to boot off of the memstick.

The OPNsense boot loader

The OPNsense boot loader looks fine. If you’re installing 17.1 using USB2 you don’t need to do anything here.

Nice feature: Early configuration importer

One notable difference from pfSense is the early configuration importer. If you have a saved configuration XML file, you can put e.g. a UFS2 filesystem on a memstick, create a directory conf on it and copy config.xml there. That makes it available in the importer.

Interface assignment

Then you have the option to assign roles to your interfaces (like WAN and LAN).

Logging into the installer

OPNsense gives you the choice to start the installer or to use a live system. Log in as user installer to perform an installation or as root in the other case. The password for both users is opnsense.

Greeting screen of the installer on the serial console

The OPNsense installer is black and white only when using the console. But that’s fine. The installer greets you with the welcome message.

Console configuration menu

The next screen lets you customize the console. You probably don’t need to do that.

Selecting the installation type

Then you need to select the installation type. You could do advanced partitioning here or setup a softraid (gmirror). We’re going with the simple installation for this post.

Choosing the drive to install on

Now you need to choose where to install to. The mSATA drive is ada0 whereas the memstick is da0.

Selecting the partition scheme to use

OPNsense also lets you choose which partition scheme to use. In case of our router this is not terribly important, especially not with our sample installation that puts everything in one partition. But since stone age is over, you might as well choose GPT anyway.

Progress bar for the installation

While the progress meter was broken with pfSense, this has obviously been fixed for OPNsense. Not that you should reinstall all that often, but still…

Installation done: Reboot!

Once the installation is finished, you of course want to reboot to your new system.

Displaying some information before rebooting

Before rebooting, OPNsense tells you how to access the Web GUI. However the IP address that it uses by default is already taken by my ISP’s modem/router box. We’re going to change that next.

Step 2: Text mode configuration

When the system has started up, you are prompted to log in. This is the default behavior which can be changed to allow unprotected login over the console like with pfSense. But in general I like that bit of extra security.

OPNsense’s text-mode configuration menu

The text-mode configuration menu looks much like that of pfSense.

Configuring the LAN interface

And the interface configuration works right the same.

Setting up DHCP on the LAN interface

As does the DHCP configuration.

Logging out and disconnecting the serial console

Since OPNsense required a login, you can also log out when you’re done. Now disconnect the serial console – we’re done with it.

Step 3: Web GUI configuration

Just like pfSense, OPNsense offers a nice Web GUI to configure all the settings. Fire up your browser on a PC that is in the same subnet (or got its IP address via DHCP from the new router) and enter the router’s LAN IP address in the URL bar.

Self-signed certificate warning

OPNsense uses https to create a secure connection, too. Of course a self-signed certificate is used which is not trusted by my Firefox. Therefore a permanent exception needs to be made.

OPNsense Web GUI login screen

Once you have confirmed the exception, you will see the login screen. Log in as root with the password opnsense.

The configuration wizard

On the first login you will be greeted by the configuration wizard. It will present you about the same choices as pfSense does (without the advertizing of the commercial version, of course).

Configuring general settings

First it’s some general information like hostname and DNS. What OPNsense offers over pfSense is i18n options: Chances are that you can configure the Web GUI to speak your language! That’s pretty nice.

Configuring time-server settings

Time server settings are just like those from pfSense.

Configuring the WAN interface

WAN configuration offers you a lot of options. Take a close look at those. Fortunately you very likely don’t need most of what is there.

Configuring the LAN interface

Same thing for the LAN configuration: You know that from pfSense.

Setting a new password for the Web GUI

Also with the password changing part there’s no surprise here.

All done. Reload the config!

That’s it. Reload the config now and you’re done with the wizard. OPNsense now has a basic configuration and is ready to be used.

Alternative installation methods

OK, you don’t have a cable to connect to the USB2 pins but you want OPNsense? There are several things that you can try. I’ve documented my attempts (including several solutions) on the OPNsense forums in case anybody needs them.

Here are a few things that you can try:

  • Install from SD card (I didn’t try that but it should indeed work)
  • Install 16.7 from USB3 with increase boot_delay and then update
  • Install 17.1 using a USB cdrom, manually enabling the console and importing a pre-made configuration

Should you install 16.7 using a USB3 port, press ESC before the loader countdown runs out. This will drop you to the loader prompt. Then enter the following:

set kern.cam.boot_delay=10000
boot

That did the trick and made the system boot up for me. The actual installation is quite similar to what I covered above.

You could also use a USB cdrom to boot the installation – of course use the OPNsense cdrom ISO in this case! However the cdrom image does not have the serial console enabled by default. So escape to the loader prompt, set some variables to enable the serial console and boot:

set boot_multicons=YES
set boot_serial=YES
set comconsole_speed=115200
set console=comconsole,vidconsole

This will work, too. But there’s one little problem with that: The TTYs are configured on their own using a configuration file – and they are not ready for serial connection! Since this is a CD, we cannot really do much about that. What we can do, however, is using the configuration importer. I will upload a basic configuration xml and add it to this post when I next install a clean OPNsense.

What’s next?

The next post will be pfSense vs. OPNsense! It will discuss some of the notable differences and when to use which one.

Building a BSD home router (pt. 4): Installing pfSense

Part 1 of this article series was about why you want to build your own router, and how to assemble the APU2 that I chose as the hardware to build this on. Part 2 gave some Unix history and explained what a serial console is. Part 3 demonstrated serial access to the APU and showed how to update its firmware.

This post is about the serial installation of pfSense, one of two FreeBSD-based router/firewall operating systems that we’re going to explore in this series (the other being OPNsense). As pfSense is the older and more established product, we’re beginning with that one.

Preparation

We’re just doing the installation here. A closer look at using pfSense or a comparison with OPNsense will be another post. Getting pfSense up and running is really easy, even when you’re using the serial console. The first step is the actual installation. In a second step you need to configure the LAN interface and then you can use the WebGUI to do the final setup.

The first thing to do, however, is getting preparing an installation medium. Head over to pfSense’s Download site. What you want is an install image for amd64. Then select USB Memstick Installer which let’s you choose the console type – obviously get the serial one!

Then get a USB stick that you can spare and dd the image on it. Once you have that ready, plug it into the APU. Next attach the serial cable to your APU and to another computer. Then connect to the console (how to do that was described in the previous post). Now power on the APU.

Step 1: Installation

Even if there’s already an OS installed on your mSATA drive, the memstick should take precedence when it comes to boot order. So you can probably just wait until the installer comes up.

pfSense’s loader menu: screwed up over the serial console…

Don’t be scared when you see garbage displayed on the screen. This is just the bootloader that’s screwed up badly when used over a serial connection (they’ve already fixed that in the beta version for the upcoming pfSense 2.4). Either just wait 10 seconds for it to boot automatically or press enter to boot right now (if you need any other options, you might want to get an ISO for pfSense, too, and test it in a VM or get a VGA image, put that on a stick and try it out on hardware that provides a local console over a screen and keyboard).

…but once the kernel loads, text is fine

As you can see, it’s only the loader. As soon as the kernel takes over, the text is displayed correctly. That means you can actually read the messages in case anything goes wrong here. If you don’t do anything, the installer will eventually come up automatically.

First screen of the installer

In the first screen of the installer you can configure the console. Most likely the defaults will be fine, though.

Selecting the installation method

Then you need to choose the installation method. We will do a quick installation but you could also do a custom installation or setup gmirror (mirrored software RAID).

The usual “this will erase your data” warning

Since installing pfSense means destroying any data that might currently be on the drive, the installer warns you that it will erase it.

Installation progress bar

If you confirmed the warning, the actual installation starts (but the progress meter is kind of useless as it seems… It remained at 5% for a while and then jumped to 100% for me).

Kernel selection

The next thing to do is to select the right kernel. Since our APU2 is a headless device, make sure that you select the embedded kernel! Otherwise you won’t be able to use the serial console with it.

Another progress bar

After the kernel is installed, the installer runs a script to do some final tasks.

Reboot message

When all is done, it’s time to reboot the system.

pfSense rebooting after installation

Just before it reboots, pfSense prints some important information on the screen, telling how to log into the WebGUI. Remove the memstick now or the APU will boot off of it once more an you’ll just see the installer again.

Step 2: Text mode configuration

The OS has been successfully installed, but leave your serial console attached for now.

pfSense’s text mode management menu

Once the system has booted, you will see the management menu. It offers a lot of tools including going to a shell (option 8) and doing everything you like. We want to configure the IP address for our LAN interface (option 2):

Configuring the LAN interface

I’m assigning 192.168.2.1 since my modem/router (yes, I’m not replacing it just yet and will operate the new router between that box and my actual network for now) has already taken 192.168.1.1. It’s not like I need a full /24 subnet for my network, but I go with that subnet mask for now.

Configuring DHCP for the LAN interface

Since I intend to use DHCP for my network, I enable a DHCP server for the LAN interface. The range of DHCP addresses that I use here is just an example for this test installation. I will cut it down to about 10 when I do my final setup. The reserved addresses before the DHCP range serve a purpose, though – more on that in a separate future post.

Back at the menu

As soon as everything is ready, you can now end the serial connection and remove the cable. We have a valid IP address on the LAN interface now after all.

Step 3: WebGUI configuration

So now we can access the WebGUI simply by entering the IP address in the URL bar of any browser. Of course the computer that runs the browser have an IP address that is on the same subnet. So you might want to change your address if that is not the case – or fire up the dhclient, it should get an address in the range that you specified (or simply reboot if your computer is configured for DHCP).

Self-signed certification warning

It’s a good thing that pfSense uses TLS so you can access the router securely via https. However the certificate it uses is self-signed and thus unknown to your browser which will display a warning. That doesn’t mean that it’s useless. In our case it’s just necessary to create an exception to accept that cert permanently.

Logging into pfSense’s WebGUI

You’ll then see the login screen. Use the username admin and the password pfsense to log in.

Running the configuration wizard

Once you’re logged in, pfSense suggests that you run the configuration wizard – and that makes sense.

A little advertising for pfSense Gold

The first screen of the wizard is an advertisement for the commercial version of pfSense called pfSense gold. If you are a company looking for more than the free “Community Edition” of pfSense will give you, have a look at this service. Maybe it’s for you.

General information configuration

First you configure some general settings like the hostname, domain, etc.

Time Server configuration

Next is the configuration of the time zone and NTP daemon.

WAN configuration

Then the WAN interface needs to be configured. There are a lot of settings there and very likely you don’t need all of them.

LAN configuration

After that comes the LAN interface. Here you can only configure the IP address and subnet mask (which we already did in text mode).

Changing the password for the WebGUI

Finally we’re prompted to change the password which is a good idea of course. Even if the WebGUI is only accessible from the LAN interface by default, it’s a matter of principle.

Configuration done: Reload!

That’s it, the wizard is finished. Time to reload the configuration.

All done, pfSense is ready

We’re done here, pfSense is installed and the basic configuration has been applied. There’s another little advertising here which is legit for a free product, I guess. We’re going to take a look at the main WebGUI and its many, many options in another post.

What’s next?

The next blog post will detail the installation of OPNsense, another excellent option for your router.

Building a BSD home router (pt. 3): Serial access and flashing the firmware

Part 1 of this article series was about why you want to build your own router, and how to assemble the APU2 that I chose as the hardware to build this on. Part 2 gave some Unix history and explained what a serial console is.

In this post we will prepare a USB memstick to update the BIOS and connect to the APU2 using the serial console. Then we’ll flash the latest firmware on there.

Cables and serial connections

In the ol’ days you would simply connect the COM port on one machine to the COM port on the other. Today a lot of newer laptops don’t even have a serial port (if yours still has one of those funny devices that you’d access through /dev/fd0, chances are pretty high though, that it also has a COM port!). Fortunately USB to serial adapter cables exist, solving that problem.

The APU2 has a male DB9 (9 pins) serial port. RS-232 is the common standard for serial communication. According to it, some pins are used to transfer information while others are used to receive information. Now if you connect two machines with a straight serial cable, both will talk on the same pins and listen on the same pins. So both will send data over pins that nobody listens on and never receive anything on the other pins. This is not really useful. To make the connection work, you need a crossed-over cable (a so-called null modem cable or adapter). This means that the receiving pins on one end are paired with transmitting pins on the other end and vice versa.

I thought that I would never need a nullmodem cable at home. I still don’t think that I might ever need a straight serial cable. And I could in fact take one home from work and return it the other day. However I could already see what would have happened in that case: When I get some time to tinker with my APU it will be on the weekend and I won’t have the cable in reach when I need it. So I got my own. And while I was at it, I decided to not only get a USB to RS-232 DB9 serial adapter cable (look for the PL2303 chipset translating USB to serial: It’s well supported across a wide range of operating systems including FreeBSD). I also bought a null modem adapter and a gender changer. So now I’m completely flexible with my gear. However you probably just want to get a null modem cable USB/female DB9 (or ask somebody who has one if you could borrow it).

Another thing that you have to know is the baud rate (modulation rate) for your connection. A higher baud rate means a faster connection. As long as both connected machines agree on the baud rate, everything is fine. If they disagree, this can lead to displaying garbage instead of the actual console or in seemingly nothing happening at all.

To flash or not to flash

At the time of this writing, PC Engines have released 5 updates for the APU2’s firmware and if you like the improvements, it makes sense to put the newest version on there. They recommend booting TinyCore Linux and then using flashrom to flash the BIOS.

Flashrom is available for FreeBSD, too. I would imagine that it works as well. However I have no experience with that and flashing stuff always bears the risk of bricking your device (for which case PC Engines offers a small rescue device). If I had thought about this right at the beginning, I would probably have tried it out. But my APU2 is already updated and since this post is public and not just for me… Well, let’s just do this by the book and use Linux for that.

If you’re a little anxious and don’t feel well about flashing at all, leave it be; in general the old BIOS will do, too. Flashing according to a guide does not have a high risk of bricking your device but even a small risk is a risk. Disclaimer: You decide. And it’s all your responsibility, of course.

Alright, we need to prepare a USB stick with TinyCore and the ROM on it. PC Engines even offer a howto guide showing how to create this using FreeBSD. That guide works, but it clearly shows that those guys know Linux a lot better than (modern) FreeBSD. For that reason I’m going to modify it slightly here and use today’s tools.

Preparing the BIOS updater memstick

First we’re going to download the TinyCore tarball and the zipped ROM (you might want to take a look if a newer version than shown here is out) and install the syslinux package (containing the that Linux bootloader):

% su -
# mkdir -p /tmp/apu2 && cd /tmp/apu2
# fetch http://pcengines.ch/file/apu_tinycore.tar.bz2 http://pcengines.ch/file/apu2_v4.0.7.rom.zip
# pkg install syslinux

Now attach your USB memstick to the system and after a couple of seconds look at the dmesg if you don’t know what device it will be attached to:

# dmesg | tail
[...]
da0 at umass-sim0 bus 0 scbus2 target 0 lun 0
da0: < USB DISK 2.0 PMAP> Removable Direct Access SPC-4 SCSI device
[...]

So for me it’s da0 in this case and I need to supplement the “X” that I use in the following commands with “0”. You should use whatever number you figured out. PC Engine’s howto suggests zeroing out the stick and if it isn’t a new and unused one that makes sense. In my case this leads to an error:

# dd if=/dev/zero of=/dev/daX bs=1m
dd: /dev/da0: Operation not supported

# mount
[...]
/dev/da0s1 on /media/disk (ext2fs, local, nosuid)

Looks like this is because GhostBSD automatically mounted da0s1 when it found an EXT2 filesystem from a previous task in there. So let’s unmount that and try again:

# umount /media/disk
# dd if=/dev/zero of=/dev/daX bs=1m

Depending on the speed and size of your memstick (and the generation of your USB port) this can take quite some time to finish. And since dd normally is working quietly, you might want to know how far it came so far. Fortunately FreeBSD implements the SIGINFO signal. Just press CTRL+T while it is running and it print some status information like this:

load: 1.52  cmd: dd 92084 [physwr] 379.15r 0.00u 0.56s 0% 3188k
2512+0 records in
2512+0 records out
2634022912 bytes transferred in 379.208179 secs (6946113 bytes/sec)

When it’s done, we can create an MBR partitioning scheme on the now empty stick as well as an MBR partition and write the bootcode on the stick so that we can boot off of it at all:

# gpart create -s mbr daX
da0 created
# gpart add -t fat32 daX
da0s1 added
# gpart bootcode -b /boot/boot0 daX
bootcode written to da0

The next thing to do is putting a FAT32 filesystem on the partition and force install the Syslinux bootloader there that will be chainloaded by the bootcode that we wrote into the MBR.

# newfs_msdos /dev/daXs1
newfs_msdos: trim 40 sectors to adjust to a multiple of 63
[...]
# syslinux -if /dev/daXs1

Now we need to mount the new filesystem and put the OS on there:

# mount -t msdosfs /dev/daXs1 /mnt
# tar xjf apu_tinycore.tar.bz2 --no-same-owner -C /mnt

We’re writing to a FAT filesystem – and as the primary filesystem from a once popular single user OS it does simply not support concepts like file ownership. That’s why we need “–no-same-owner” here (otherwise we’d see harmless but unnecessary warnings). As the next step we’ll add the ROM image and check its integrity – we don’t want to flash garbage on our APU and brick it, do we?

# unzip -d /mnt apu2_v4.0.7.rom.zip
# grep -c `md5 -q /mnt/apu2_v4.0.7.rom` /mnt/apu2_v4.0.7.rom.md5
1

Make sure that the last command outputs 1 (in which case the calculated md5 hash matches)! If it does not, delete the zip file, download it again and extract it over the corrupt ROM file. Finally sync the filesystem, unmount it and remove the USB stick:

# sync
# umount /mnt

The memstick is ready. If you want to test it, boot some other PC or laptop from it. If you can read the line “Booting the kernel” and then nothing seems to happen anymore, it means that it’s working. TinyCore is configured to use the serial console, that’s why you don’t see anything on your screen and your keyboard doesn’t do anything after that. Just turn your computer off and plug the USB stick out.

Attaching the serial console and flashing the BIOS

Alright, back to the APU2 (finally). Put the memstick into one of the USB ports and attach your serial cable to the COM port. Now connect the other end of the null modem cable with another computer running FreeBSD (or Linux or whatever – this should all work, you’ll just have to figure out how to connect to a serial console on your OS).

Open a terminal, become root and attach the serial console (cuaU0 is the USB to serial adapter, 115,200 the baud rate):

% su -
# cu -l /dev/cuaU0 -s 115200
Connected

Now connect the APU2 to power and see what happens! If you don’t do anything, the BIOS should load TinyCore from the memstick after a couple of seconds:

Connected to the serial console – and booting Linux

If nothing happens, you might have the wrong cable (is it really crossed-over?). Or maybe you’ve mistyped the baud rate? The “Connected” by the way only means that your host system has attached to the serial cable. You’ll get that message even when the other end is not connected to anything or the machine at the other end is turned off.

Once Linux was loaded, use flashrom to update the APU’s firmware like PC Engines show in their howto. Then reboot.

Preparing to flash the BIOS

The APU2’s BIOS supports the serial console. That means that even before the machine has booted an operating system with serial console capability, you can access and configure the BIOS of the headless machine:

Serial access to the BIOS settings

Another nice thing that comes with the APU is the Memtest program. If you want to know whether your new hardware is actually good or might probably have bad ram, put it to the test for a couple of hours or over night:

The firmware comes with Memtest

If you’re using cu as in my example, you can close the serial connection using the character sequence ~. (tilde and dot).

What’s next?

You now know how to access your box using the serial console. Next time we’ll make use of that skill again to put pfSense as the first of two options on the APU. The other option is OPNsense which will be covered in a later post. Both are FreeBSD-based router/firewall operating systems.