FreeBSD package building pt. 2: Basic Synth

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/freebsd_package_building_pt2.gmi

The previous article was an introduction into package building on FreeBSD in general. It also included references to various other articles about package management and working with ports. Familiarity with those topics is assumed here. If you don’t feel confident, have a look there and do a bit of reading first. Or just go ahead and do some research if you hit something that you have trouble understanding – package building is not a beginner’s topic but it ain’t rocket science, either.

In the following section I’m following a “naive approach” of just installing and exploring Synth on a typical desktop system. Afterwards I’m switching to the test system that was described in the previous post so that I can show updating, too.

Synth basics

So let’s get started with Synth now, shall we? The first thing to do is of course installing it:

# pkg install synth

If you want to or need to build it from source instead, the origin is ports-mgmt/synth. It has one compile-time option: Enabling or disabling the watchdog feature. By default it is enabled which means that Synth is supervising the build processes and if one looks like it has stalled, the watchdog will kill it. When you are building on very slow hardware (why are you doing this in the first place?) it may be an anti-feature that causes very complex ports (e.g. Firefox) to be killed even though it would otherwise eventually finish. In that case you can disable the watchdog option and rebuild Synth. Or you go to the junkyard and get some more capable hardware!

Synth port options

As with every new piece of software, it’s probably a good idea to get an overview of what it can do:

% synth help
Summary of command line options - see synth.1 man page for more details
===============================================================================
synth status              Dry-run: Shows what 'upgrade-system' would build
synth configure           Brings up interactive configuration menu
synth upgrade-system      Incremental rebuild of installed packages on system.
                          Afterwards, the local repository is rebuilt and the
                          system packages are automatically upgraded.
synth prepare-system      Like 'upgrade-system' but ends when repo is rebuilt
synth rebuild-repository  Rebuilds local Synth repository on command
synth purge-distfiles     Deletes obsolete source distribution files
synth status-everything   Dry-run: Shows what 'everything' would build
synth everything          Builds entire ports tree and rebuilds repository
synth version             Displays version, description and usage summary
synth help                Displays this screen
synth status [ports]      Dry-run: Shows what will be rebuilt with given list
synth build [ports]       Incrementally build ports based on given list, but
                          asks before updating repository and system
synth just-build [ports]  Like 'build', but skips post-build questions
synth install [ports]     Like 'build', but upgrades system without asking
synth force [ports]       Like 'build', but deletes existing packages first

synth test [ports]        Just builds with DEVELOPER=yes; pre-deletes pkgs

[ports] is a space-delimited list of origins, e.g. editors/joe editors/emacs.
It may also be a path to a file containing one origin per line.

That’s a nice list and even better: The commands are explained in a way that mere mortals can understand. Basically it can compare installed program versions to those in ports for you with synth status. You can configure synth with synth configure (we’ll look at this later, the default config is fine for now). And it can bulk-build packages and use those to upgrade the system for you with synth upgrade-system. And those are the important ones that you’ll need to begin with.

There’s also synth prepare-system in case you want Synth to only build the packages (maybe you want to do the upgrade manually later). With synth rebuild-repository you can make the tool, well, rebuild the repository from the currently built packages. Since it does this after finishing package builds, you will need it only if you cancelled building somewhere in the middle and want it to rebuild the repo anyway. The synth purge-distfiles command is useful if you use Synth for a while. It will scan for no longer needed distfiles (for obsolete program versions) and potentially free quite a bit of space for you.

And that’s it. The synth status-everything and synth everything commands are only useful if you want to build packages for the entire FreeBSD ports collection. But that’s certainly not basic usage! The various options that act on a single port or a list of ports is also advanced usage. They’ll come in handy if you plan on developing ports, build packages for other machines to use or have special needs. If you plan to keep your machine’s installed packages consistent for production use, know that these commands exist but stay away.

Package status

So much for some theory and on to actually doing something. Let’s ask Synth about the status of our packages:

# synth status
It seems that a blank PORTSDIR is defined in /etc/make.conf
Configuration failed to load.

Alright, this message can be a little confusing as by default there is no such file in FreeBSD! What Synth actually wants to tell you is that it cannot find a ports tree on the system and thus cannot do anything. So let’s get the ports in place right now:

# portsnap auto

With the ports tree available on our system we can try again:

# synth status
Configuration invalid: [C] Distfiles directory: /usr/ports/distfiles
Rather than manually creating a directory at this location, consider
using a location outside of the ports tree. Don't forget to set
'DISTDIR' to this new location in /etc/make.conf though.

This time the problem is pretty straight-forward: The distfiles directory does not exist. Synth gives us some advice in case we want to use a custom location, however I prefer the standard one.

# mkdir /usr/ports/distfiles
# synth status
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
 progress: 3.32%

Depending on your build machine this can take anything from a couple of minutes to a quarter of an hour or so. Before FreeBSD introduced port flavors, it was a matter of seconds, but for a few years now Synth has to do some additional work up-front. When it’s done it will print something like this:

Querying system about current package installations.
> Installed package ignored, devel/py-evdev package unmatched
> Installed package ignored, devel/py-pyudev package unmatched
> Installed package ignored, devel/py-six package unmatched
> Stand by, comparing installed packages against the ports tree.
> Stand by, building pkg(8) first ... done!
> These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
>   N => print/indexinfo
>   N => devel/gettext-runtime
> [...]

For the first run, Synth has to build all of the packages. Therefore all are marked new. But what’s that “Installed package ignored” thing? That usually happens if a package is installed on the system that was built from a port that does no longer exist in the new tree. Sometimes it’s also weirdness that can happen with flavored ports. Disregard this if it’s only affecting packages that you don’t want to use directly. The correct ones will be pulled in as dependencies anyway. Now let’s build our first package set:

# synth prepare-system

After gathering some information, Synth will start building the packages that are currently installed on this machine (which for simplicity’s sake resemble those of the test system). Let’s have a look:

Synth started building packages

By default, Synth shows a nice curses-based text UI that displays a lot of information (see screenshot above). It shows the total count of packages to build, the number of packages that remain to be built and the count of already built packages. In case a port does not successfully build on this system (update ports and try again – if the problem persists either file a bug report or contact the maintainer), Synth displays that, too. Ignored packages are those that don’t work on your particular system; maybe the application is known to not build on the version of FreeBSD you are using or not on your architecture (if for example you’re building on ARM). Skipped count goes up if a failed port was a dependency for others. Obviously Synth cannot build them.

Synth also displays the current system load: A load of e.g. 2.0 means that two cores of your CPU are completely busy. If your CPU supports Hyper-threading that basically doubles the available core count. A load higher than 8.0 on a 4 core system with HT means that the system has currently more work than it’s able to fulfill concurrently. It’s not ideal but it’s not something to worry too much about, either. Watch swap usage. A little bit of swapping for large ports is not much of a problem. But if your system swaps a lot that will slow down the package building a lot. Should you manage to run out of swap, package builds will fail. You can adjust Synth’s configuration if you’re unhappy with either load or swap usage. But we’ll get to that.

Synth after about 20 minutes

There’s also the number of packages built per hour and the impulse as well as the elapsed time. Initially packages per hour and impulse are identical because the two are similar. However the former is the average number of packages built over the whole build time while the latter is the number of packages built within the last 500 seconds.

But that’s only the top bar. The next part of the UI is for the builders. Builders are clean build environments (“chroot jails”) that are created before building a package and then teared down afterwards. The next package gets a new fresh environment. The number of builders means how many packages can be building concurrently. On the screenshot you can see 6 builders which are color-coded to be easier to distinguish. This may look different on your machine and here’s why: Synth tries to guess reasonable defaults for your machine. If you have an old dual-core PC, it will use less than the six builders it deemed right for the quad-core i7 that I used as my test machine here. Expect to see a much higher number of builders on modern servers with a higher core count.

Idle builders after 5 hours

For each builder you see how long it has been working already, which phase it is currently in (remember the various build targets of the ports infrastructure?) which port the builder is occupied with and how many lines of log output the builder has has produced so far. Especially the last information is not going to help you a lot when you begin building packages. After a while you’ll know roughly how many lines of output some of the big ports like LLVM produce and can judge if it’s going to take another two hours to finish or more like half an hour. (And then a new version of LLVM comes out which takes even longer to build so that your previous idea of “how many lines” is no longer valid. That’s how things go.)

And finally there’s the rest of the screen made up of a list of recently finished packages. If you take a look at the second screenshot, you’ll see some ports where the origin ends with @py37. Here the builder is busy building a flavored port – and it’s building for Python 3.7. What’s @native you ask? Well, Python ports are a typical example of flavored ports but they are not the only ones. The binutils port for example is able to be built as part of a native toolchain or a cross toolchain in case you want to e.g. cross compile packages for riscv on your much more powerful amd64 machine.

First builder has shut down

What’s the deal with idle builders like those on screenshot 4? Idle builders are the ones that Synth has already prepared for another package to build in but has not been able to use so far. Take a look at screenshot 4: There’s 53 more packages to build but only two builders are occupied with work while four are idle. Can you guess why? The two ports currently building are LLVM10 and Rust. And the “problem” is that all other 51 packages that are still on the list depend on either LLVM or Rust directly or indirectly! So while there’s more work for the builders, Synth has to wait for the dependencies to finish building before it can start building the remaining packages.

At some point no further builders will be required. In that case Synth shuts them down (see screenshot 5).

Last package build starting after about 6 hours

When Synth has finished building all the packages requested, it will present you with the final statistics. Then it cleans up the package directory, removing obsolete old packages. Eventually it will build a valid pkg(8) repository for you – and then it’s finally done.

Synth has completed its task

Should you ever want to quit a package run early, try to avoid CTRL-C. If you press CTRL-Q instead, Synth will shutdown gracefully. This means that now new builders will be started and the tool exits properly once those that are already running complete.

Pkg(8) repositories

I’m covering the case where you want to use your own package repository instead of the official FreeBSD one. If you want to use both, make sure you read what I pointed my readers to in the previous article. Then configure your new repository as I do here but simply don’t disable the official repo.

The standard FreeBSD repository is configured in /etc/pkg/FreeBSD.conf. Do not make changes there, though! This file belongs to the base system and upgrades may overwrite it. If you want to change the settings, create /usr/local/etc/pkg/repos/FreeBSD.conf (and the missing directories). The latter file will override the former. In case you just want to disable the official package repository, simply put this single line into the file:

FreeBSD: { enabled: no }

Synth should automatically generate a config file there if you use it to upgrade the system. The file is called /usr/local/etc/pkg/repos/00_synth.conf and has the following content:

# Automatically generated.

Synth: {
  url      : file:///var/synth/live_packages,
  priority : 0,
  enabled  : yes,
}

Now you only need to update the repository information:

# pkg update

And that’s it. Pkg(8) will now happily use your local package repository.

What’s next?

The next article will feature Synth configuration, upgrading and advanced usage.

FreeBSD package building pt. 1: Introduction and test system

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/freebsd_package_building_pt1.gmi

In 2017 I started to post a series of well-received articles that was meant to culminate in a tutorial on building packages on FreeBSD. I wrote about the history of package managers, about how to use FreeBSD’s pkg(8) package manager (part 1 and part 2) as well as an introduction to working with ports (part 1 and part 2).

Then I had to stop for technical reasons (after a major change to the ports infrastructure, the tool that I wanted to write about had not been updated to work with the new ports tree, yet!). In 2019 I eventually found the time to publish two more articles that were originally meant to come right after the 2017 posts. They covered using classic tools for ports management (part 1 and part 2).

They were meant to be the stepping stone to what I actually wanted to cover: Package building! The second article ended with:

I’ve planned for two more articles that will cover building from ports the modern way(tm) – and I hope that it will not take me another two years before I come to it…

That was… Yikes! In fall 2019… So I’d better hurry up now. I had hinted that using ports directly was not the recommended thing to do anymore and that while you should use packages unless you need ports, you really should use packages even in the latter case! And that’s what we’re going to do in the next few articles.

Why roll your own packages?

There are valid reasons to not use FreeBSD’s official packages:

  • Most frequently you need build-time options configured differently.
  • Or your organization uses a different default version of a scripting language (Perl, Python, Ruby, …) that what FreeBSD chose at the time being.
  • Maybe you’re running a customized version of FreeBSD that’s not officially supported.
  • Perhaps you need programs where the license forbids binary redistribution.
  • Or you use custom ports for some programs that of course cannot be available in official packages then.

Things like that. People choose ports when they need more control over their applications. There are good reasons to avoid using ports the traditional way, too, however:

  • It takes time and resources to build everything from source; especially if you do this on multiple machines, it’s a waste.
  • It makes updates much more complicated (see the second 2019 post mentioned above).
  • It clutters your system with build-time dependencies that need to be installed first so that the actual programs can be built.
  • Depending on what other software your machine has installed, the resulting programs might differ from that of other machines even if they built the same port!

While the former two points are mostly relevant if you manage multiple machines, I’d recommend rolling your own packages even for the single FreeBSD workstation that you use – if the official packages don’t suit you. Let me state this again: Try to go with the packages provided by the FreeBSD project first. Build your own packages if you have to (educating yourself is a completely valid reason to do it anyway, though).

Package builders

When you decide to roll your own packages, you have two options: Synth, the much easier and nicer package builder and Poudriere, the advanced build tool that FreeBSD uses and provides official documentation for.

Which one should you choose? I’m going to show how to work with both so you can compare them and make an informed decision. If you’re just getting started, you may want to give Synth a try. It is also a good choice when you use DragonFly BSD, too: The dsynth tool that they have in base was inspired by Synth (and if it wasn’t written in Ada they certainly would just have imported it instead of creating a re-implementation in C). You should also know that the Synth project is in maintenance mode. Its author still fixes bugs and takes pull requests on GitHub, but it’s feature-complete.

The main advantage of Synth is that it’s dead simple to setup and use, too, whereas Poudriere is a somewhat complex beast. Synth also shines when you want to use it to keep one machine up to date with packages as it can do those updates for you. Poudriere on the other hand allows you to do things like maintaining package repositories for multiple versions of FreeBSD as well as multiple architectures from one build machine. If you need that, forget Synth.

Ports and Git

One major change that was made in FreeBSD since the previous article was published is that the project migrated to using Git repositories. FreeBSD development started on CVS but in 2008 the src repository was successfully migrated to using Subversion. In mid 2012 docs and ports also made the switch. Subversion has been used ever since until December 2020 when doc and src transitioned to Git. After some remaining issues were solved, ports also migrated to Git in April 2021. While src changes get pushed back to Subversion for FreeBSD’s branches of 11 and 12, when it comes to ports, an era has ended.

Get rid of that habit of visiting svnweb.freebsd.org and start getting used to working with cgit.freebsd.org instead.

If you are unsure of your Git skills, you may want to at least skim over the Git-primer in FreeBSD’s documentation.

At least get familiar with the concepts. You should for example know that Git is able to do things like a shallow clone; looking things up when you need them is no problem but not being aware of them at all is.

While both Subversion and Git are used for version control and both use repositories, they are fundamentally different. Subversion is the best known version control system of the second generation (so-called (centralized) “networked VCS”). Git is the most popular one of the third generation (“decentralized VCS”).

If you use Subversion, there’s a central repository somewhere and you checkout a certain revision of the versioned files from there. The files that were checked out form your working directory. In Git you clone a remote repository and thus receive all the data needed to get a local copy of the whole repo. The files for the working directory are checked out from the local copy. The major difference is: You have the whole history available locally.

In case you have no need for all that data, you do a shallow clone instead of a regular full clone. To give you an example: If you do a shallow clone of the current ports tree today, the result is about 840 MB in /usr/ports – of which 85 MB in size is the Git repository. A full clone is about 1.7 GB in size of which about 920 MB is for the repo! So if you don’t need the history, save some space on your drive and save some donated bandwidth of the FreeBSD project.

Considerations before you start

While you can certainly start rolling your own packages on a fresh system, it’s also fine to begin doing so on a system that has used the standard FreeBSD ports or packages so far. There’s nothing wrong with that actually. Taking the opposite way and going back from custom packages to the official ones is also possible, of course. That latter case requires some more planning, though. Think about why you started building your own packages in the first place, find out if the standard packages fit your current needs and think about the consequences. If you’re uncertain, you may want to start over with regular packages. If you want to make the switch back, it’s best to re-install all the packages (use the flag -f with pkg upgrade). If you’re using ZFS in a way that supports Boot Environments, create one first.

In fact you can even choose to use both official and custom packages on the same machine: Instead of building the full set of packages that you need, you build just the packages yourself that you need to customize and use the packages from the standard FreeBSD package repositories for the rest. This works by configuring your package repository as an additional one for pkg(8) and giving it a higher priority. If you consider doing this and using multiple package repositories, be sure to

% man 5 pkg.conf

first. It doesn’t hurt to skim over the whole manpage, but make sure that you at least read the section REPOSITORY CONFIGURATION. I used that approach on a weaker machine in the past, but wouldn’t generally recommend it. If your hardware is recent enough, you can compile everything yourself. Otherwise it might make more sense to build on a somewhat beefy system and distribute the packages to your other machine(s). But that’s only me. Maybe mixing packages is the right solution for your use case.

Building a test system

The rest of this post is about building a test system to simulate an environment with pre-installed packages. I do this so that I can show off a few things regarding updates in the next post. If you want to follow along, make sure that you have Git installed and that your directory or dataset /usr/ports as well as /usr/src is empty. I’m assuming that you’re using FreeBSD 13.0.

The first thing to do is getting the ports tree in place. I’m doing a shallow clone of the 2021 first quarter branch (so I can update later). Then I clone operating system source repository as well:

# git clone --depth 1 -b 2021Q1 https://git.freebsd.org/ports.git /usr/ports
# git clone --depth 1 -b releng/13.0 https://git.freebsd.org/src.git /usr/src

Now we don’t need the packages anymore and forcefully remove them all (including pkg). The reason for this is that we want to build the older versions of the old ports tree that we just cloned.

# pkg delete -af

For convenience we’re going to build all the required programs using portmaster as discussed in a previous article (see the beginning of this post). Of course we need to build and install it first:

# make -C /usr/ports/ports-mgmt/portmaster install clean

Alright. Now we’re building some leaf ports to simulate a very simple development system. These ports draw in a number of dependencies so that we end up with about 350 packages (which is still a very low count for a desktop system). I’m building tmux and sudo simply because I always need to have them available and git just because we needed it before anyway. The rest is the graphics drivers, a simple subset of X11 + the command to set a different keyboard layout as well as the awesome window manager and the simple but nice GTK+ terminal emulator called sakura. Since I’ll need to take screenshots for the upcoming posts, I figured that I might include an application for that, too.

# portmaster -dG sysutils/tmux security/sudo devel/git graphics/drm-kmod x11/xorg-minimal x11/setxkbmap x11-wm/awesome x11/sakura x11/xfce4-screenshooter-plugin

And that’s it for today.

What’s next?

Next station is taking a look at building packages and updating the system using Synth.

FreeBSD router take 2 (pt. 4): Demoting my ISP’s router

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/bsd_router_take_2_pt4.gmi

Since I built my first OPNsense-based router, it had been a secondary router only. Its “WAN” port was connected to my ISP’s modem/router box which dealt with establishing the actual Internet connection and acting as the gateway and DHCP server for my OPNsense. In other words: It has only ever been a second line of defense for my LAN network behind it.

Also since I started building said first router of mine I had the goal of eventually replacing my ISP’s router with it instead of just adding a secondary device. In retrospect it still was the right choice to start with a first step like I did. It allowed me to play with OPNsense and get familiar with it while I did not fully depend on it working right all the time. It has been fun and the fact that OPNsense never broke for me when updating was a reassuring experience.

Promoting OPNsense to be my primary router

The following image shows what my network looked like up until a short while ago (red = my ISP’s property, blue = OPNsense, green = transparent networking devices):

Diagram of my network before the change

As you can see, my ISP’s modem/router was the device directly connected to my ISP’s line, forming my primary network behind it (LAN 1). Occasionally I’d plug a laptop or something into that directly. The APU running OPNsense was a secondary router behind which the network for my regular devices began (LAN 2). It meant that I could better tune packet filtering rules than the primary router allowed me. I could do proper QoS and other things. But it only was an additional device and didn’t really obsolete the one that I actually wanted to get rid of.

I did a bit of research and finally in early April I went ahead and made the switch. For my network OPNsense is in charge now! But I didn’t actually get rid of my ISP’s old box just yet. Why? Well… because of IP telephony. This is a topic of its own and I hadn’t been inclined to doing too many things at once. So I decided to try out if my phone would still work if I demoted my ISP’s router to be the secondary router and would let it manage telephony. I just set the device to operate in client mode, connected the IP phone, tried to call my parents – and was very relieved to find that it just worked!

So this is the new network diagram:

My network after the change

The APU is not technically capable of connecting with my ISP; I needed a DSL modem for that. The model that I chose operates in bridge mode so that it’s network-transparent but let’s OPNsense establish the connection. Directly connected to that additional device is the APU which now also handles a second network segment via its OPT1 interface: My old router is connected to that.

At the end of the day I have one more device involved but packets originating from my main computers no longer have to go through two routers. VoIP packets have to now, but at least the primary router is the box that I control, so that’s an improvement as well.

Choosing a DSL modem

After my research, I settled on the Zyxel VMG1312-B30A which is marketed as a “Wireless N VDSL2 4-port gateway with USB”. It’s an older device from 2012, but it’s still sold. While the specs don’t look very impressive today, I don’t care about which wireless standard it supports and such. I got it for another feature that it offers: Bridge mode.

Zyxel VMG1312-B30A DSL modem top view

If I were to simply replace my ISP’s device with this one, it’d even be a downgrade – the Zyxel makes an even worse router than what I had. But operating in bridged mode it simply terminates the DSL circuit of the telephone line and communicates using the DMT (Discrete Multitone Modulation) protocol with the ISP’s DSLAM (Digital Subscriber Line Access Multiplexer).

Zyxel VMG1312-B30A DSL back and bottom top view

Before settling on a modem, do some research on which technology is being used for DSL in your country! For my Zyxel device there are two separate variants of the same model: One for the so-called Annex-A and one for Annex-B. The former specifies “DSL over POTS” (= Plain Old Telephone Service) while the latter is for “DSL over ISDN”. Both devices are physically different, so be sure to get the right one (in Germany for example it’s Annex-B)! Annex-A uses the smaller RJ-11 jacks to connect to the ISP line while Annex-B uses the standard RJ-45 jacks that are also used for ethernet cables.

Taking a first look at the modem

By default the modem operates in router mode and has the IP 192.168.1.1/24 assigned. Configuration is accessible via a web UI on ports 80 and 443. The user is admin and the password 1234. Configure a workstation a static IP in the same subnet and connect it to the device, then login.

The modem has a standard overview page called “connection status” and four more sections that offer a menu each. First one is Network Setting.

“Network Setting” menu

The most important pages in that menu are Broadband, Wireless, Home Networking and Power Management.

Menu number two is called Security.

“Security” menu

This time there are two interesting pages: Firewall and MAC filter.

The third menu is “System Monitor”.

“System Monitor” menu

There you will find logs, the ARP table, the routing table and so on if you need it.

Finally there’s the “Maintenance” section.

“Maintenance” menu

The most important pages here are User Account, Remote MGMT, Firmware Upgrade and Configuration.

You should probably start by updating the firmware to the latest available version, but I’ll go through some of the pages here in order.

Modem configuration

First go to Network Setting -> Broadband. Set the device to operate in Bridge mode. Depending on what ISP you are using, you might need to have to set a VLAN tag for the connection to work. In my case choosing VLAN 7 is required. You might need to do some research or try out some possibilities.

Make the modem work in bridge mode

Then go to Network Setting -> Wireless. Do yourself a favor and just disable it. It will save you some power and offer additional security. If you need to update the firmware again or make another change, just physically connect a machine to it.

Turning off wireless access

Next is Network Setting -> Home Network. Turn off the DHCP server there. And if you’re paranoid, assign it a different IP – preferably in a different private address range.

Disabling the DHCP server

Lastly for the first section go to Network Setting -> Power Management. Here you can turn off everything that you don’t need. I only left the WAN port as well as one LAN port active and chose to unpower the rest.

Unpowering the LED and most ports

Next is Security -> Firewall. Since we’re not using Router mode, the firewall doesn’t make any sense. Off it goes.

Switching off the Firewall

For a bit of extra paranoia go to Security -> MAC Filter. Here you can choose to allow access to the modem only from certain NICs. If you consider doing this, make sure that you understand the consequences. Allow a minimal of two MAC addresses to not lose access if the respective NIC / machine should ever get damaged. If you only configure one, make sure to at least write it down and deposit it somewhere safe in case you need to spoof it. Otherwise you’ll have to factory-reset the modem when you managed to lock yourself out.

MAC Filter interface

Definitely go to Maintenance -> User Account and change the default password to something stronger.

User Account settings

Pay Maintenance -> Remote MGMT a visit. Turn off everything that you don’t need. You definitely don’t want Telnet, FTP or plain HTTP. Chances are that you don’t want SNMP either (if you do want to have it you know why). Disable ping if that makes you more feel better. And when it comes to SSH, here’s the reason I turned it off:

Unable to negotiate with [IP ADDRESS] port 22: no matching exchange method found. Their offer: diffie-hellman-group-sha1

This means that they ship a version of OpenSSH from 2015 or older (and probably never updated it since 2012 – if they even used the most current one back then). You can make your client talk to it anyway, but for me there’s generally no need for it.

Remote MGMT choices

Definitely go to Maintenance -> Firmware Upgrade and do it now if you haven’t done so already.

Firmware Upgrade

Finally there’s Maintenance -> Configuration. Here you can backup the configuration settings you just made and download an archive to your computer. Doesn’t hurt to do that.

Configuration

So much for the modem. There’s more things it can do but they are mostly only relevant in router mode (and sometimes even then only when you have special requirements).

OPNsense dialup

With the modem fully configured and working, it’s time to configure OPNsense to do the DSL dialup. I chose to rename the first network interface from WAN to PPPoE, but that’s only a name. You need to go to Interfaces -> PPPoE (or whatever yours is called) and change the IPv4 Configuration type from DHCP to PPPoE (unless you have an IPv6-only line of course in which case you’d configure that instead).

OPNsense WAN connection configuration

Further down enter the username and password for the PPPoE connection. Check the documents you got from your ISP, they should be on there somewhere. If they aren’t, ask for them.

OPNsense PPPoE configuration

And that’s all. Save your changes and if everything is correct, OPNsense will do the dialup and establish an Internet connection! Much better now that a trusted device does this, isn’t it?

Conclusion

My new setup is not perfect. Ideally I’d make my OPNsense machine deal with the IP telephony, too. Before even attempting that I will however need to do a lot of reading upfront. So there’s another long-term goal.

Nevertheless this was a change for the better. I made another step in reclaiming my own network. So far I’ve been running this setup for a month and did not face any problems. There has been a short power outage once: After power was back, the APU and the modem booted and before long OPNsense had re-established the connection and I was online again.

What’s next?

The next article will be about building custom packages on OPNsense (since it’s a somewhat involved topic it will probably be split into two posts, though).

FreeBSD router take 2 (pt. 3): Excursion – De-hardening OPNsense for 2022?

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/bsd_router_take_2_pt3.gmi

After OPNsense announced that they would rebase on vanilla FreeBSD instead of going on with HardenedBSD, I wrote the previous article on what “security” means when it comes to networked devices that are connected to the Internet. It also took a look at the fields where FreeBSD is doing pretty good. There’s also the other side of the coin however. Being a person who really likes FreeBSD and enjoys working with it, this article is not one that I looked forward to writing. But FreeBSD is not all sunshine and roses (who would have thought?). And people should be aware of that to make an educated choice. So here we go.

Defaults / (lack of) exploit mitigations

It’s the last two points from the previous article’s list that FreeBSD admittedly sucks at. As discussed there, you can make your FreeBSD systems a lot more secure than they are after a default installation. But FreeBSD does not believe in a “one size fits all” security concept. Truth be told, I’d also be very skeptical about such a concept. Either it will be so general that it’s not worth even laughing about or it will be highly theoretical and probably collide with real deployments pretty soon.

FreeBSD provides a lot of documentation to help you understand the system. However only you know your specific use case and therefore only you can put together the best possible security concept. I strongly support this way of thinking. It is true however that FreeBSD could do a lot better with regards to basic security. Why isn’t a firewall active by default? If it was that would probably be ipfw with a simplistic rule set. One of the first things that I’d do on a new installation would be disabling it and configuring pf instead. And if somebody really has a use case where there should be no firewall – well, nobody would stop you from just disabling it. Coming with any firewall enabled by default would neither limit your choices nor would it make FreeBSD unfit for any sensible scenario. Making a change like that has POLA implications, though (FreeBSD’s “Policy of least astonishment”). But if the will was there, a way would be found.

There’s other things that make it really hard to come up with an excuse for. Frankly speaking: When it comes to mitigation techniques, FreeBSD is hardly a modern operating system. Linux, Windows – basically every common system you can name did a whole lot of work in that area over the last two decades. In comparison FreeBSD did almost nothing. Unfortunately there is not much more to say about that.

And one of the few things that were done, wasn’t done right according to some security researchers. FreeBSD claims to support ASLR (Address space layout randomization). It’s not enabled by default, but it’s there:

# sysctl -a | grep aslr
kern.elf32.aslr.stack_gap: 3
kern.elf32.aslr.honor_sbrk: 1
kern.elf32.aslr.pie_enable: 0
kern.elf32.aslr.enable: 0
kern.elf64.aslr.stack_gap: 3
kern.elf64.aslr.honor_sbrk: 1
kern.elf64.aslr.pie_enable: 0
kern.elf64.aslr.enable: 0
vm.aslr_restarts: 0

The idea of this mitigation technique is to randomly arrange data of processes in memory to make it harder for an attacker to hit a targeted function. In general ASLR’s effectiveness as a mitigation has been doubted by a lot of people who are into security. Then again, it does not come with a high cost and so it’s often seen as a baseline of protection. It’s been standard in OpenBSD since 2003, in Linux since 2005 and in OS X as well as in Windows since 2007. NetBSD was a bit late to the party and only implemented it in 2009. Heck, even Oracle Solaris adopted ASLR in 2012. And FreeBSD? Totally took their time. Got it first in 12.1 which was released in late 2019. But better late than never and maybe the wait was worth it if we got a superior implementation for that?

There’s one problem, though: FreeBSD’s implementation has been criticized as half-baked… According to Shawn Webb it’s not even ASLR but ASR. So we got what we got extremely late, it’s disabled by default and even if you enable it it’s a pretty weak form of what is considered a very basic mitigation. This does not make FreeBSD look too good.

Is FreeBSD well-maintained?

Let’s poke another pain-point, shall we? FreeBSD is much, much less active in cleaning up their system than OpenBSD for example. If you take a look at the source repository, you won’t have to search too long to find things that are not all that pretty. Here’s one example of a commit updating the comcontrol manpage.

The commit removes a reference to the sio(4) interface in the comcontrol(8) manpage. This change is available in the recently released FreeBSD 13.0, older supported releases 12.2 and 11.4 still mention the interface. The thing is that sio(4) was removed from the GENERIC kernel in 2008 – which means that the manpage change that finally shipped this year (and for the very latest version only!) could easily have been shipped with FreeBSD 8.0 over ten years ago…

Want one more? Have a look here. Meet “pnpinfo” which according to the manpage “reports information about Plug-n-Play ISA devices“! Hasn’t been touched in over ten years and is very obviously completely obsolete. It’s not built by default anymore, can’t be built manually either (due to a missing system header file) – but it’s still there in the source tree. It looks like it was still part of FreeBSD 10.0 (early 2014) but removed for 10.1 (late 2014). Till the end, the pnpinfo(8) manpage referenced pnp(4) which in turn had already been removed in FreeBSD 4.6 (2002)!

Right, this is nitpicking around very minor issues. Basically every project has dusty corners and when it is the size of FreeBSD, it would be close to a miracle to not have any. Still it’s only two easy to find examples out of many that show one thing: There’s room for improvement. Plenty actually. But while *BSD prides itself in good documentation, little leftovers like this don’t have such a huge impact on security. Could really be much worse right? For example if little to no maintenance was being done on very important system components responsible for, say secure authentication?

I can hear anything from deep sighs to screams of agony from readers familiar with FreeBSD even before I put a link to Heimdal in the base system here…

Really – FreeBSD ships with Heimdal 1.5.2 in the base system… This version was released in 2012 (!!) and nobody should have trouble believing me that there’s a bunch of nasty CVEs for it. Right, everybody knows that you should never use kerberos from the base system. If you need it, always install either security/heimdal or security/krb5-$version from ports or packages. That way you’ll get versions that are up to date. But honestly: Why the heck is that ancient base system version even there? Nobody should have used it in almost a decade! What’s the point in having a trap like that lurking in base? To see if unsuspecting users might fall into it and make an acquaintance of the poisoned spears at the bottom? That’ll teach them a valuable lesson, eh? No, sorry. No point in even trying to whitewash this. It is just hideous and a real disgrace.

And then there’s of course the recent turmoil around the flawed Wireguard implementation that almost made it into FreeBSD 13. If you don’t know what I’m talking about, consider skimming over this Ars Technica article.

It is actually not a good article and I expected more of Jim Salter; he does a podcast called “2.5 admins” together with FreeBSD developer Allan Jude and they discussed the topic a couple of days before Salter wrote the linked article, forgetting some of the important things and concentrating on minor matters to have “a good story”… It will introduce you to the drama, however. Keep in mind that Jim pretends that the flawed code was “probably” only removed because the original Wireguard inventor intervened even though FreeBSD developers were looking at the code and there already were people unhappy with it (whereas he denies that the even more recent happenings around Linux and the University of Minnesota showed that things in Linux world are also far from perfect).

Bottom line: There’s all kinds of problems in FreeBSD. From small cosmetics to heavy-duty stuff. But FreeBSD is an Open Source project. If you think about contributing fixes (even for the very simple things): By all means do so! It’s not that FreeBSD wants to be in the sorry state it is in regarding certain areas. The project is taking new contributions with open arms. You’d help make the world a little bit better for many people. And there’s plenty of valuable skills to acquire if you choose to go down that road. Doc committers in FreeBSD are equal in their rights to ports and source committers, by the way. If you’ve got a bit of time for it and an interest in tech (you’re reading articles like this not because you don’t care at all, do you?) seriously consider it.

HardenedBSD

At this point the sunshine that the previous article may have shown is probably gone and there are some pretty dark clouds in the sky. Don’t let the problems that I pointed out here scare you away. Remember that the above was written by a FreeBSD user – not a former user. Everything added up, FreeBSD is a decent platform that’s not worse than any other. In fact it has a lot of advantages that help accepting some of the disadvantages. Be aware of the ugly part, though. It might bite you otherwise.

But is this a god-given situation that we cannot do anything about? Is it either the really nice features and sane structure of FreeBSD or better mitigations but much less overall usefulness of OpenBSD (alternatively the better mitigations but the chaotic mess that is Linux these days)? Fortunately not: Enter HardenedBSD.

Have a look at this image to get an idea of what HardenedBSD is doing:

HardenedBSD feature comparison

It’s only four security features listed there that OpenBSD has but HardenedBSD doesn’t. Of course the comparison is not complete, missing out a several good things in OpenBSD like e.g. pledge. However HardenedBSD also has a lot that go even further than what OpenBSD does.

And that’s really, really impressive. Keep in mind that HardenedBSD is basically FreeBSD with a ton of security improvements to it: It has ZFS, jails and all the good stuff. It’s a bit less convenient to use (e.g. you will have to understand additional tools like secadm to toggle certain mitigation features on or off for specific programs). It offers you the means to make system administration a fair bit more cumbersome – while making life terribly hard for attackers. If you are serious about security and accept that there is no free lunch, you’re willing to endure the additional restrictions for a huge gain in hardening your system.

HardenedBSD is a hardened but not a hard fork of FreeBSD. It tracks upstream FreeBSD and merges new code from there. The project also aims to develop security features outside of FreeBSD but to ultimately give the changes back. This would be a huge gain for security-focused FreeBSD users. A very small project however has also very little chance of getting FreeBSD to accept proposed hardening techniques. For that reason HardenedBSD needs every bit of support it can get.

For some time, HardenedBSD also had LibreSSL in base instead of OpenSSL. They had to switch back for the simple reason that the team was to small to keep up with the work required for such an invasive change along with all the other security improvements. And now that OPNsense has announced to ditch HardenedBSD, it will lose some more badly needed support.

So is it a hopeless case? Well, not quite. OPNsense was definitely the most prominent user of HardenedBSD but certainly not the only one. There are people and companies using it. There is being research done with it (see the e.g. this bunker jails article).

Co-founder Shawn Webb also managed to get a foundation started for it and even to attract an impressive amount of donations last year. I’d say that $13,000 instead of 11,000 they had aimed for is not bad at all! Especially if you compare it to the NetBSD foundation which only managed to get about 24,000 of their 50,000 goal even though they are a much older and bigger project.

I’ve been thinking about using HardenedBSD instead of FreeBSD when I build my next workstation. I’ll probably also use it when I reinstall my server and see how that goes. Both will probably things to write about here on the Neunix Gemlog.

Does leaving HardenedBSD make (OPN)sense?

Decisions like this are always a tradeoff and I’m not under the impression that the OPNsense team made this one without carefully considering the matter. In short-term I think that tracking mainstream FreeBSD will definitely benefit OPNsense. Here’s a couple of reasons:

  • It makes development easier in general
  • It will speed up adoption of newer releases
  • It will free resources (e.g. currently the team has to backport fixes to a no longer supported FreeBSD release)
  • It makes debugging easier
  • It might attract additional contributors familiar with FreeBSD but not HardenedBSD

Sounds good, right? If you’re willing to sacrifice the additional hardening of HardenedBSD it sure does. And I think that most people would in fact prefer to go down that route.

IMHO OPNsense is hurting itself in the long run, though. The major reason for ditching HardenedBSD is that it is too much of a niche platform after all. With OPNsense leaving it, it will become even more niche. It is a very important project to eventually take FreeBSD into the right direction. Let’s not underestimate the gem that we have here! Trying to increase adoption would be what we should be doing, not decreasing it further.

But I don’t want to challenge the decision that has been made, write a petition and bring unrest to the community. What OPNsense needs is to continue evolving for the better. One goal that aligns perfectly with the new strategy is getting rid of some more quirks that OPNsense inherited from pfSense and rather doing things like FreeBSD does. This would benefit everybody.

And who knows: Perhaps we’ll see something like “HardenedSense” in the future? Not as a fork but as a community build for people who prefer to stick with a hardened system for their packet filter needs. I hope that this is food for thought for some readers. Maybe we can start a discussion over at the forums or so. If there’s anybody interested in this, please let me know.

Why not OpenBSD?

Following the announcement of OPNsense to part ways with HardenedBSD, some users over on Reddit proposed to rebase on OpenBSD instead. Let’s consider this for a moment.

OpenBSD is generally regarded as a very, very secure operating system. It has a great lot of mitigations in place, a nice and clean codebase and a reasonable-sized community. That’s certainly appealing. People also frequently mention that it has a much newer version of Pf which would be very much beneficial for a project like OPNsense.

There’s a couple of reasons why this is not as good an idea as it seems, though. I actually like what the OpenBSD people are doing. No, truth be told, I admire their security first stance and the fact that they are willing to take it to the extreme anytime. But… Exactly this makes it the wrong choice for anything like OPNsense:

  • Performance is not a primary goal for OpenBSD. If you want a top-notch router, you’ve ruled it out.
  • OpenBSD is a research OS. You can use it in production but you have to live with things like NO ABI stability whatsoever.
  • There’s a lot less software packaged for OpenBSD.
  • OpenBSD does not provide safe data storage, they don’t have any next-gen filesystem.

Let’s also address the misconception of “newer Pf on OpenBSD”: This is not true. Pf originated in OpenBSD when they dropped (due to licensing issues) IPF which they used before and replaced it with their own packet filter. Pf was later ported to FreeBSD (and NetBSD). After those ports happened, OpenBSD continued to improve Pf. One thing that they did was revising the syntax. FreeBSD did not sync their Pf with OpenBSD anymore – but for a good reason! They had improved their version of Pf to make it perform much better with multi-core CPUs. Contributing those changes back to OpenBSD was hopeless since OpenBSD was largely not SMP-capable at that time. For that reason Pf on OpenBSD and Pf on FreeBSD diverged, up to the point where merging newer changes from OpenBSD was simply not feasible anymore.

So it’s not that OpenBSD has “newer PF” – it’s more like both OpenBSD and FreeBSD have distinct versions of Pf that are actively developed but are quite different despite the common name. Rebasing OPNsense on OpenBSD would not give the users a much better Pf. In fact the major user-visible advantage of OpenBSD’s version of Pf – i.e. the simpler syntax – would not even be user-visible on OPNsense as people use the GUI to create their rules! It would on the contrary mean that code changes would be required so that the OPNsense application responsible for the rules would generate the rules in the new syntax expected by OpenBSD’s Pf.

There would also be a lot of other things to change. OpenBSD’s networking works quite a bit differently (e.g. the system’s hostname goes into /etc/myname instead of into /etc/rc.conf as used in FreeBSD). The init system is slightly different. Packaging works very differently (not using /usr/local for example and the package managers are simply worlds apart). And so on.

Conclusion

FreeBSD is a solid operating system that’s doing well overall but is severely lacking in certain areas. HardenedBSD offers all the benefits of FreeBSD without a lot of the weaknesses and is an innovating force when it comes to strong security. OPNsense leaving HardenedBSD behind is a sensible choice considering OPNsense alone but a very unfortunate move for the FreeBSD ecosystem as a whole. OpenBSD is not the right base for OPNsense either.

If you care for FreeBSD and security, please support HardenedBSD. Let’s keep it going strong – maybe there’s the chance of having a community edition of 22.1 and onward that’s still going to be based on HardenedBSD if there is enough interest.