Advance!BSD – thoughts on a not-for-profit project to support *BSD (2/2)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/advance_bsd_pt2.gmi

[Edit (22/07/21)]Yet another poll on Reddit asking for the preferred means of communication ended with great support for continuing the project in a Subreddit of its own. So after today the previous part of my double post was featured on the popular BSDNow! podcast, I finally went ahead and started r/AdvanceBSD. It’s now open for discussion. Let’s see what will come from this![/edit]

The previous article provided an introduction of the Advanced!BSD project idea by providing some background as well as discussing the “why”. This article focuses on the “what”.

Shared services or virtualization?

Another general question is what makes more sense to focus on: Some form of virtualization (either full-fledged VMs or paravirtualization like FreeBSD’s jails either with or without VNET) or rather providing shared services? There certainly is a need for both. And while I’m certain that the vast majority of BSD people would be perfectly capable of renting a VM and doing all the things they need themselves, I’m not sure if everybody wants to.

Thinking about what my own use cases would be, the result is likely a mixture of both. If by accident a) the shared service was built upon exactly the software components that I prefer, b) there was a nice interface providing the level of control that I need and c) somebody else kept the whole thing nicely up to date, I wouldn’t say no! If however some service is provided by something that I loathe (let’s say, exim or even sendmail for email!), I’d rather do things myself.

Depending on what preferences people have, the focus of the project might be either way. Personally I’d like to see some high-quality shared services at some point. But that doesn’t necessarily mean the project would have to start with that.

Which services?

It may look like a logical decision to start with domains. For practical reasons I’d rather stay away from that for the beginning, however. Here’s why: Domain business is hard. The profit margin is very low even for the big players – and you need to have a lot of domains in your portfolio to get decent prices. Before that you’re losing money if you want to compete with the established providers. In addition to that, it’s not exactly simple to properly automate things like domain transfers that can keep real people occupied otherwise. Considerations like this don’t make that field particularly attractive.

DNS is probably a much better field to start with (provided we can get at least one person on board who’s experienced in advanced DNS topics like e.g. DNSSEC and DANE). The initial time to invest into proper configuration of nameservers is not to be neglected but much more on the doable side compared to domain registration and migration for multiple TLDs. And since for DNS you always at least need two servers, this would also be a great one to make use of two different BSD operating systems right from the start!

Plain Web hosting would certainly be the easiest thing to start with. However you definitely want to support TLS certificates with LE today – and supporting wildcard certs for domains requires DNS to be ready for it. Also intermediate hosting is a bit more complex already: A lot of people will want PHP. Some require Python. Others still want Perl. Throw in things like mod_security & friends and the topic gets a lot more complicated pretty quickly. Plus: What webserver to use? Will OpenBSD HTTPd suffice? Should it be Lighty? Nginx perhaps? Or do people need Apache HTTPd?

Email is a huge topic. We’d have to investigate anti-spam and anti-virus software. We’d need to build up reputation for our IPs so that the major mail providers would take email messages from us. And – of course – we should get DNS working first for things like the MX records, SPF, …

Another possibility would be offering complete “products” like a Gitea instance, an XMPP chat server, the Sogo groupware, etc. This only makes sense if we find out that by chance there are a lot of people interested in the same services. In general on the net I’d say WordPress is a sure bet. But my guess is that many people who might be interested in a project like this rather use Hugo, Jekyll and the like.

Each of those topics would of course need to be discussed further if actually a real group forms to make Advance!BSD happen.

Involving neighbors and others?

While I have no doubts that the BSD community is big enough for such a project to succeed with, it might still make sense to connect with some other communities that are also “niche” (i.e. underrepresented due to mainstream exclusivism) but nevertheless important or interesting.

One “neighbor” that comes to mind is the illumos community. While I’d clearly like to focus on BSD, I’d be open to let them participate if they wish. Eventually supporting illumos, too, if people who are capable of providing it join in, would not be a bad thing. We can even offer to give all the revenue that comes from illumos hosting back to that community. It wouldn’t be a loss for us since customers who want that wouldn’t have booked a BSD jail / VM / service in the first place anyway. At the same time it would further diversify the project and help it grow. So there could be a mutual interest.

I’m personally interested in the Gemini project (see FAQ link at the top of this article if you don’t know what it is). As an effort within the so-called “Small Internet” it’s not an incredibly huge community, yet. Due to its nature it’s a pretty technically versed one, though. There’s probably a majority of Linux users there, but taking it into consideration could be for mutual benefit again! Running a gemini server and maintaining a site in Geminispace in addition to the Web that informs about Advance!BSD and the services provided, is really not much hassle. It might well get the project some wider recognition, however. And if we’ve got that Gemini server, anyway, we might consider Gemini hosting as a bonus provided for free to our customers (if they want it). It’s very low on resources and Gemini people would get another free hosting service while we’d benefit from them spreading the word. Win-win.

If there’s enough interest regarding Gemini in the future, we could even decide to dive all in and provide a means of interacting with the services provided by an experimental interface that uses the Gemini protocol! We’d have to explore how to do something like this. Gemini is deliberately a simple protocol, but people have built search engines with it, there’s a wiki system and so on. One nice thing is that client certificates are in common use with Gemini and are often supported by the browser applications. That’s a pretty nice way of proving your identity either without the need for a password of as a second factor.

Just one more example: The POWER platform. It has a much lower market share than amd64 or aarch64 obviously, but there are some people who like it quite a bit. Hosting on ARM is nothing too exciting anymore these days, but hosting on POWER certainly is. We might consider platform diversity as another field that people could select to fund with their money. I can imagine that some people who are not necessarily into *BSD would consider using our services (initially on amd64, of course!) if the money would eventually be used to purchase a POWER 9 server.

Hosting on *BSD running on POWER would certainly be experimental for some time to come, but enthusiasts probably wouldn’t mind that too much. This could be a great chance for the BSD operating systems to improve support on that architecture and it would also be a nice opportunity to diversify the POWER ecosystem. In other words: Another win-win situation (if there’s enough interest in such a thing).

How to get this started?

Well, I’m not going to claim that I now the best way or even have a solid master plan. But here’s some steps that come to mind when I think about it:

Phase 0: Figure out if there’s any interest in something like this at all. Check – there is!

Phase 1: Determine what means of communication would work best for most people interested in the project: A Subreddit? Classic mailing list? IRC / some messenger (which one)? A Web forum? Something else?

Phase 2: Form an early community, collect ideas and thoughts that people have and discuss them! Consider problems that are likely to arise.

Phase 3: Decide on project governance and for example found a core team.

Phase 4: Try to reach some kind of consensus on the concrete goals of the project as well as which are short-term, mid-term and long-term.

Phase 5: Ask people interested about their areas of proficiency, make a list of who can do what. In parallel come up with a list of necessary tasks, then find people who can do the actual work and get things going.

Phase 6+: Let’s not decide on this before phase 2 ends!

Requirements

There’s two kinds of resources that we need server-related (both hardware and software) and person-related (skills as well as some free time). Regarding servers, any reasonably new machines will probably do. When it comes to the skill sets of people required for the project, things become a bit more complicated. Here’s just a few roles that would need to be filled:

  • We obviously need people who install machines, keep them up to date and make sure services continue to run (by setting up proper monitoring and paying attention to notifications) – i.e. administrators.
  • Very early on we need web developers: While it is theoretically possible to only interact with customers via email, this is absolutely not a feasible solution. There needs to be some kind of platform where people can login and view / change their services. Also without a somewhat respectable website it will be hard to find any people who’d actually book any service.
  • DevOps people. We live in the age of “automate or die!”. Also proper automation does not only mean less manual work but also fewer human mistakes.
  • Supporters. We need people who agree to dedicate a bit of time every week to look into problems people have, find a solution to them and interact with the customers via mail.

And so on.

Project identity

I’d like the project to follow some kind of theme early on. It should convey “yes, we are BSD!” and as long as it’s about a community-driven project, a bit of humor would certainly be appealing to people. We have the daemon and the fork that kind of symbolize BSD as a whole. Which means that we could either use those or try to fit all the specific BSD mascots into the picture. The latter might be harder to do without making it look too artificial and affected. So what about Project: Trident? Just kidding of course! 😉

Here’s one proposal: The mascot could be the polar beast: A daemon riding a wild-looking polar bear in a knightly fashion with a long fork in hand as a lance, a small orange pennant tied to the latter and a halo over its head. Thanks to the bear it’s clear that he’s at the north pole. This theme would allow for a couple of jokes like this: “Pretty cool but not cold(-hearted)” [the south pole has much lower overall temperature] and “Best of all: No clumsy flightless birds here!” 😉

We should make clear though, that at the end of the day we’re all friends in Open Source and despite regularly making fun of each other are not hostile towards Linux but want to go our own way.

Things like that. I’m sure that people come up with all kinds of funny themes if they give it a try.

What’s next?

I’m hoping that this provides some more food for thought and that we’ll be able to establish some kind of communication channel that people interested in this will use to really start discussing things. Regarding my usual articles I’ll probably write about Poudriere next.

Advance!BSD – thoughts on a not-for-profit project to support *BSD (1/2)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/advance_bsd_pt1.gmi

There are multiple reasons why I am a BSD user and enthusiast. For one thing I enjoy using systems where the design benefits from the holistic approach (whole OS vs. “kernel + some packages”). I’ve come to appreciate things like good documentation, preference of simplicity over unnecessary complexity and permissive licensing. There’s a different spirit to the community that makes me feel more at home. Well, and truth be told: I do have a heart for “outsiders” who are doing well but get far less attention due to the towering popularity of a certain other system. In addition to that I’m very much concerned about the new de-facto monopoly in open source.

While the broader BSD community is far less fragmented than what you know from the Linux ecosystem, it’s also much smaller. Considering the little manpower that the BSD projects have, they are doing an outstanding job. But a lot of people seem to agree on the fact that due to the small amount of resources available, the BSDs are pretty far away from maximizing their full potential.

There are in fact a lot of people out there who’d like to help improve the situation. But coordination of volunteer effort is hard. Linux is what Linux is for a substantial part due to corporate funding. And while there are also companies that support BSD, the question is: Could we probably do better?

A (nonprofit) BSD-first service provider?

After thinking about this for quite a while, I finally just asked on Reddit what other people think about such a project – and I’ve been blown away by the response!

While I had hoped that most people would agree that this could be an interesting thing or even consider supporting it, I had not anticipated that the most popular option on the poll would be the one where people express their interest in the project with the prospect of perhaps participating actively in getting it started!

20 votes (after only one day) for maybe participating in the project!

A lot of projects struggle for years to find people who might be willing to join. With projects that weren’t even started yet, and thus have nothing to show off, it’s even harder to get some attention. Getting 20 people to support the cause in just one day was quite a surprise for me. Sure, that’s only a poll vote on Reddit and thus completely without any obligation. But let’s assume that 1/4 would actually join such a project and contribute – 5 people is not bad at all. Depending on what skills they bring in, even 2 or 3 might suffice to get something started, further increasing the odds that more people join in. The hardest part in getting a project on they way is finding people who are willing to pioneer it and make something that sounds interesting actually work well.

Why “Advance!BSD”?

The name is just a working title, BTW. I won’t insist on it and am in fact pretty sure that something better could easily be found.

Recently there have been two longer discussions (also on Reddit) about what the BSDs lack to be more competitive. There were a lot of great ideas and I’m one of the people who’d like to see at least some of them being implemented eventually. But let’s be realistic: This is not very likely to happen. There’s enough work going on regarding the must-haves, and when developers decide to work on a nice-to-have, it’ll be something that they want. If we really want to see some things not of very high priority to the various projects eventually land, we need to take care of that ourselves. If you’re a developer with the required skills and enough free time: Great! Nobody can really stop you. If you aren’t: Too bad.

I’ve thought about the possibilities of crowd-funding that we have today. In theory we could try to find a enough people who’d like to see a feature and are willing to spend some money so that the group could contract a developer. Even though I believe that there are enough developers out there who’d love to do some paid work on *BSD and would probably even prefer such a task over one that pays better, I’m not very optimistic about it. There’s not such a high chance of finding people who want the same feature implemented and would be willing to pay for it. Certainly not over a longer time which would be required to collect enough money. Also I don’t really like that begging for people’s money approach, either. So I came up with a different idea:

If you like *BSD you’re almost certainly an IT person. IT people all have a couple of things in common: We love tech. We use email. Almost everybody has his or her own homepage. Which means: We need domains. We need DNS. We need Webspace, etc. And we get that from service providers. Some of which are somewhat BSD-friendly, many of which are not. And even cloud providers for example that offer BSD as an option usually don’t do that because they love it (and regularly that shows…).

So what about providing services instead of just asking for money? Imagine starting a “hosting club” where like-minded people get something rolling that works for them and that could at some point be turned into a not-for-profit hosting provider that lives and breathes BSD! The latter would use the money acquired to pay the running costs and spend the rest on improving the BSDs. What could that look like?

Well, for example like this: Members could vote for a general topic for each year (like e.g. “desktop”, “drivers”, “de-GPLing”, “porting efforts”, …) and propose (as well as discuss) concrete project ideas over the year then voting again at the end of the year when the available money is to be spent. Any project that is beneficial for more than one specific BSD gets bonus points.

Which BSD operating system(s)?

In a follow-up poll I asked people interested in the project about which BSDs they are proficient with. Considering the general market share within *BSD it came to no surprise:

The largest group of votes were for FreeBSD somewhat closely followed by OpenBSD. People voting NetBSD, DragonFly BSD or multiple followed only after a huge gap.

Most people favor FreeBSD and OpenBSD

Pretty early on one person commented that even though NetBSD was his favorite OS, it might make most sense to go with FreeBSD for such a project for example due to jails being a very useful feature for that. I can imagine that there might be more people who decided to put their personal preferences aside and vote for what they think makes most sense for this use case.

To be honest, I’d like to see all of the BSDs being part of this in some way or another. But to be realistic, I think we need to start with something. This should be one or two BSD systems that project members are familiar with. Right now FreeBSD and OpenBSD seem to be the reasonable choices here. Of course we need to take into account what those systems excel in and which one to use for what.

Problems with the not-for-profit status

Like with everything, there’s pros and cons regarding the aim for this to be a not-for-profit eventually.

Pro:

It ensures that nobody involved in it could ever become greedy and try to sabotage the original idea of providing money to support *BSD. It would also protect such an organization from becoming attractive for buyout by a for-profit competitor should it go well. There would be benefits regarding taxes. And I’d imagine that it gives a good feeling to the customers which could be turned into a competitive advantage.

Contra:

The price to pay is inflexibility. A not-for-profit can donate money only to another not-for-profit organizations – and that very likely only in the country that it was formed in. With e.g. the FreeBSD foundation and the OpenBSD foundation we have to potential organizations that we might want to donate to; however one is US-based while the other one is Canadian. A for-profit company is free to spend money however it wishes. There might be other limitations that I’m not aware off. Going with not-for-profit would require consulting lawyers upfront.

A federated model?

One interesting idea that came up on Reddit was that of possibly going with a federated model. In this case project members / supporters who own a server but don’t need all of its resources would dedicate some percentage of CPU, memory and disk space for a VM that could be used for the project. The user who suggested this envisions that there could be some sort of market place where people can offer those resources to people who want a VM. The person who donates the resources gets to decide what field the biggest part of the money made from this would be spent on. The customer on the other hand can also influence the the direction by picking e.g. VM a) where the money goes to desktop improvements over option b) were he’d support permissively licensed alternatives.

I like this idea and think that we should discuss it. It has the obvious benefit of being able to start without somewhat high upfront costs for hardware. Also more people might be ok with a “donate some spare resources” approach for machines that are running anyway. The downside certainly is: Reliability. Donated resources might basically be withdrawn anytime. What could / should we do about that?

What’s next?

While this part 1 mostly covered the “why”, part two of the article is more about the “what” and the “how”.

FreeBSD package building pt. 5: Sophisticated Synth

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/freebsd_package_building_pt5.gmi

In the previous posts of this series, there was an introduction to package building on FreeBSD and we discussed basic Synth usage. The program’s configuration, working with compiler cache and using the tool to update the installed applications was covered as well. We also discussed Synth web reports, serving repositories over HTTP and building package sets. We also took a brief look at the logs to find out why some ports failed.

In this article we are going to sign our package repository and explore make.conf options. We will also add an additional profile to build packages for an obsolete FreeBSD release that is 32-bit and automate package building with cron.

Changing the structure

So far we’ve only considered building basically one local package set (even though we’ve shared it). If we want to have a single build server manage multiple package sets, we will need a somewhat more complex directory structure to allow for separate repositories and such. Should you be building on a UFS system, it’s easy: Just create the additional directories. I’m using ZFS here, though, and need to think if I want to create the whole structure as several datasets or just two single datasets with custom mount? As usually there’s pros and cons for both. I’m going with the former here:

# rm -r /var/synth
# zfs create zroot/var/synth
# zfs create zroot/var/synth/www
# zfs create zroot/var/synth/www/log
# zfs create zroot/var/synth/www/log/13.0_amd64
# zfs create zroot/var/synth/www/packages
# zfs create zroot/var/synth/www/packages/13.0_amd64
# synth configure

Now we need to adapt the Synth configuration for the new paths:

B needs to be set to /var/synth/www/packages/13.0_amd64 and E to /var/synth/www/log/13.0_amd64. Normally I’d create a custom profile for that, but as I’m covering that a little later in this article, we’re going to abuse the LiveSystem default profile for now.

Next is re-configuring the webserver:

# vi /usr/local/etc/obhttpd.conf

Remove the block return directive in the location “/” block on the synth.local vhost and replace it with:

directory auto index

Then change the location to “/*”. I’m also removing the second location block. Create a new .htpasswd and bring over authentication to the main block if you want to.

# service obhttpd restart

Repository signing

To be able to use signing, we need a key pair available to Synth. Use the openssl command to create a private key, change permissions and then create a public key, too:

# openssl genrsa -out /usr/local/etc/synth/LiveSystem-private.key 2048
# chmod 0400 /usr/local/etc/synth/LiveSystem-private.key
# openssl rsa -pubout -in /usr/local/etc/synth/LiveSystem-private.key -out /usr/local/etc/synth/LiveSystem-public.key

Mind the filenames here! The LiveSystem part refers to the name of the profile we’re using. If you want to sign different repositories resulting from various profiles, make sure that you place the two key files for each of the profiles in /usr/local/etc/synth.

While you’re at it, consider either generating a self-signed TLS certificate or using Let’s Encrypt (if you have own a proper domain). If you opted to use TLS, change the webserver configuration once more to have it serve both the log and the package vhosts via HTTPS. There’s an example configuration (obhttpd.conf.sample) that comes with obhttpd in case you want to take a look. It covers HTTPS vhosts.

Alright! Since we changed the paths, we don’t currently have any repository to sign. Let’s build a popular browser now:

# synth build www/firefox

Firefox failed in the configure phase!

Firefox failed to build. This is what the log says:

DEBUG: Executing: `/usr/local/bin/cbindgen --version`
DEBUG: /usr/local/bin/cbindgen has version 0.18.0
ERROR: cbindgen version 0.18.0 is too old. At least version 0.19.0 is required.

Please update using 'cargo install cbindgen --force' or running
'./mach bootstrap', after removing the existing executable located at
/usr/local/bin/cbindgen.

===>  Script "configure" failed unexpectedly.
Please report the problem to gecko@FreeBSD.org [maintainer] and attach the
"/construction/xports/www/firefox/work/.build/config.log" including the output
of the failure of your make command. Also, it might be a good idea to provide
an overview of all packages installed on your system (e.g. a
/usr/local/sbin/pkg-static info -g -Ea).
*** Error code 1

Stop.
make: stopped in /xports/www/firefox



--------------------------------------------------
--  Termination
--------------------------------------------------
Finished: Friday, 11 JUN 2021 at 03:03:06 UTC
Duration: 00:03:06

Oh well! We’ve hit a problem in the ports tree. Somebody updated the Firefox port in our branch to a version that requires a newer cbindgen port than is available in the same branch! Breakage like this does happen sometimes (we’re all human after all). What to do about it? In our case: Ignore it as it’s only an example. Otherwise I’d advise you to update to a newer ports tree as these problems are usually quickly redeemed.

Synth is asking whether it should rebuild the repository. Yes, we want to do that. Then it asks if it should update the system with the newly built packages. And no, not now. Also note: The synth build command that we used here, is interactive and thus not well fit if you want to automate things:

Would you like to rebuild the local repository (Y/N)? y
Stand by, recursively scanning 1 port serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Would you like to upgrade your system with the new packages now (Y/N)? n

What else do we have to do to sign the repository? Nothing. Synth has already done that and even changed the local repository configuration to make the package manager verify the signature:

# tail -n 3 /usr/local/etc/pkg/00_synth.conf
  signature_type: PUBKEY,
  pubkey        : /usr/local/etc/synth/LiveSystem-public.key
}

That wasn’t so hard now, was it? You might want to know that Synth also supports using a signing server instead of signing locally. If this is something you’re interested in, do a man 1 synth and read the appropriate section.

Global options with make.conf

FreeBSD has two main configuration files that affect the compilation process when using the system compiler. The more general one, /etc/make.conf and another one that is only used when building FreeBSD from source: /etc/src.conf. Since we’re talking about building ports, we can ignore the latter.

There’s a manual page, make.conf(5), which describes some of the options that can be put into there. Most of the ones covered there are only relevant for building the system. Do by all means leave things like CFLAGS alone if you don’t know what you’re doing! Regarding the ports tree, it’s most useful to set or unset common options globally. It’s very tedious to set all the options for your ports manually like this:

# make -C /usr/ports/sysutils/tmux config-recursive

You need to do this for specific ports that you want to change the options for. But if there’s some that you have a global policy for, it’s better to use make.conf. Let’s say we want to never include documentation and examples in our ports. This would be done by adding the following line to /etc/make.conf:

OPTIONS_UNSET+=DOCS EXAMPLES

This affects all ports whether built by Synth or not as well as each and every Synth profile. Let’s say we also want no foreign language support in our packages for the default Synth profile (but in all others), we’d create the file /usr/local/etc/synth/LiveSystem-make.conf and put the following in there:

OPTIONS_UNSET+=NLS

That setting will build packages without NLS in addition to building without DOCS and EXAMPLES – if “LiveSystem” is the active profile.

If you want to build all ports that support it with e.g. the DEBUG option, add another line:

OPTION_SET+=DEBUG

Some common options that you might want to use include:

  • X11
  • CUPS
  • GTK3
  • QT5
  • JAVA
  • MANPAGES

After unsetting DOCS and EXAMPLES globally as well as NLS for the default profile, we’re going to rebuild the installed packages next:

# synth prepare-system

Package rebuild done

Note that Synth only rebuilt the packages that were affected by the changed options either directly or via their dependencies. For that reason only 219 of the 344 packages actually installed were rebuilt. If we now use pkg upgrade, this is what happens (see screenshot):

pkg upgrade will free space due to removing docs, examples and nls files

Ignore the 3 packages getting updated; this is due to packages that were skipped due to the Rust failure originally. That port has successfully built when we were trying to build Firefox, so our latest package runs built three more packages that have not been updated to, yet.

More interestingly: There’s 31 reinstalls. Most of them due to the package manager detecting changed options and one due to a change to a required shared library. It’s not hard to do the math and figure out that 31 is quite a bit less than 219. It’s a little less obvious that build-time dependencies count towards that greater number while they don’t appear in the count of packages that are eventually reinstalled. Still it’s true that Synth takes a “better safe than sorry” approach and tends to rebuild some packages that pkg(8) will end up not reinstalling. But this is not much of a problem, especially if you’re using ccache.

Alternative profiles

If you want to use Synth to build more than one specific set of packages for exactly one platform, you can. One way to achive this would be to always change Synth’s configuration. But that’s tedious and error prone. For that reason profiles exist. They allow you to have multiple different configurations available at the same time. If you’re simply running Synth from the command line like we’ve always done so far, it will use the configuration of the active profile.

To show off how powerful this is, we’re going to do something a little special: Building 32-bit packages for the no longer supported version of FreeBSD 12.1. Since amd64 CPUs are capable of running i386 programs, this does not even involve emulation. We need to create a couple of new directories first:

# mkdir /var/synth/www/packages/12.1_i386
# mkdir /var/synth/www/log/12.1_i386
# mkdir -p /var/synth/sysroot/12.1_i386

The last one is something that you might or might not be familiar with. A sysroot is a somewhat common term for – well, the root of a system. The sysroot of our running system is /. But we can put the data for other systems somewhere in our filesystem. If we put the base system of 32-bit 12.1-RELEASE into the directory created last, that’ll be a sysroot for 12.1 i386. Technically we don’t need all of the base system and could cherry-pick. It’s easier to simply use the whole thing, though:

# fetch -o /tmp/12.1-i386-base.txz http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/i386/12.1-RELEASE/base.txz
# tar -C /var/synth/sysroot/12.1_i386 -xf /tmp/12.1-i386-base.txz
# rm /tmp/12.1-i386-base.txz
# synth configure

Alright. Now we’re going to create an additional profile. To do so, press > (greater than key), then chose 2 (i.e. Create new profile) and give it a name like e.g. 12.1-i386. Then change the following three settings:

B: /var/synth/www/packages/12.1_i386
E: /var/synth/www/log/12.1_i386
G: /var/synth/sysroot/12.1_i386

That’s all, after you save the configuration you’re ready to go. Create a list of packages you want to build and let Synth do it’s thing:

# synth just-build /var/synth/pkglist.12.1_i386

The build will fail almost immediately. Why? Let’s take a look. Building pkg(8) failed and here’s why:

--------------------------------------------------------------------------------
--  Phase: check-sanity
--------------------------------------------------------------------------------
/!\ ERROR: /!\

Ports Collection support for your FreeBSD version has ended, and no ports are
guaranteed to build on this system. Please upgrade to a supported release.

No support will be provided if you silence this message by defining
ALLOW_UNSUPPORTED_SYSTEM.

*** Error code 1

Stop.
make: stopped in /xports/ports-mgmt/pkg

Ok, since we’re trying to build for a system that’s not supported anymore, the ports infrastructure warns us about that. We have to tell it to ignore that. How do we do that? You might have guessed: By using a make.conf for the profile we’re using for this set of packages:

# echo ALLOW_UNSUPPORTED_SYSTEM=1 > /usr/local/etc/synth/12.1-i386-make.conf

Then try again to build the set – and it will just work.

Successfully built all the packages using the i386 profile

Automation & Hooks

Last but not least let’s put everything we’ve done so far together to automate building two package sets. We can make use of cron(8) to schedule the tasks. Let’s add the first one to /etc/crontab like this:

0	23	*	*	7	root	env TERM=dumb /usr/local/bin/synth just-build /var/synth/pkglist-12.1-i386

What does it do? It will run synth at 11pm every sunday to build all of the packages defined in the package list referenced there. There’s two things to note here:

  1. You need to disable curses mode in all the profiles you’re going to use. Synth still expects to find the TERM environment variable to figure out the terminal’s capabilities. You can set it to dumb as done here or to xterm or other valid values. If you don’t set it at all, Synth will not run.
  2. The cron entry as we’re using it here will use the active profile for Synth. It’s better to explicitly state which profile should be used. Let’s add another line to crontab for building the amd64 packages for 13.0 on Friday night:
0	23	*	*	5	root	env TERM=dumb SYNTHPROFILE=LiveSystem /usr/local/bin/synth just-build /var/synth/pkglist-13.0-amd64

In general I’d recommend to consider not calling synth directly from cron but to write small scripts instead. You could for example backup the current package set before actually starting the new build or you could snapshot the dataset after the successful build and zfs-send it off to another system.

One last thing that you should be aware of is that Synth provides hooks like hook_run_start, hook_run_end, hook_pkg_failure and so on. If you’re considering using hooks, have a look at the Synth manpage, they are covered well there.

What’s next?

Next topic would be covering Poudriere. However I’m considering taking a little break from package building and writing about something else instead before returning to this topic.

FreeBSD package building pt. 4: (Slightly) Advanced Synth

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/freebsd_package_building_pt4.gmi

In the previous posts of this series, there was an introduction to package building on FreeBSD and we discussed basic Synth usage. The program’s configuration, working with compiler cache and using the tool to update the installed applications was covered as well.

This article is about some advanced functionality of Synth like using web reports, building package sets and serving repositories as well as taking a look at logs.

Reporting

Synth comes with very useful reporting reporting functionality. For the default LiveSystem profile, Synth writes its logs to /var/log/synth. There it also creates a subdirectory called Reports and puts an advanced web report in there. It looks like this:

% ls -1 /var/log/synth/Report 
01_history.json
favicon.png
index.html
progress.css
progress.js
summary.json
synth.png

We’re going to build and setup a web server to make it accessible. I will use OpenBSD HTTPd for a reason that we’re going to talk about in a minute (besides me liking it quite a bit). Let’s use Synth to install it first. Then we’re going to enable it and create a log directory for it to use:

# synth install www/obhttpd
# sysrc obhttpd_enable=YES
# mkdir /var/log/obhttpd

OpenBSD HTTPd successfully installed by Synth

Alright. Remember how I told you not to change Synth’s directory configuration unless you have a reason for it? Now we have: We’re going to serve both the web report and the packages over http and we’re using OpenBSD HTTPd for that. That browser is chrooted: Not just by default, BTW but by design! You cannot turn it off. Unless told otherwise, Synth has a directory structure that doesn’t fit this scenario well. So we’re going to change it.

First thing is creating a new directory for the logs and then changing the configuration so Synth uses that:

# mkdir -p /var/synth/www/log
> # synth configure

choc

Change setting E to /var/synth/www/log and save. The new directory will of course be empty. We could copy the old files over, but let’s just rebuild HTTPd instead. Using synth build or synth just-build doesn’t work, though; the tool will detect that an up to date package has already been built and that nothing needs to be done. That’s what the force command is handy for:

# synth force www/obhttpd

Web server setup for reports

Now that we have something to serve, we can edit the webserver’s configuration file. Simple delete everything and put something like this into /usr/local/etc/obhttpd.conf:

chroot "/var/synth/www"
logdir "/var/log/obhttpd"

server synth.local {
    listen on * port 80
    root "/log"
    log style combined
    location "/" {
        block return 302 "http://$HTTP_HOST/Report/index.html"
    }
}

This defines where the chroot begins and thus where the unprivileged webserver processes are contained. It also defines the log directory as well as a virtual host (simply called “server” in OpenBSD HTTPd) with the name “synth.local”. Either replace this with a proper domain name that fits your scheme and configure your DNS accordingly or use /etc/hosts on all machines that need to access it to define the name there.

The server part is pretty simple, too. It makes HTTPd bind on port 80 on every usable network interface. The web root is defined relative to the chroot, so in this case it points to /var/synth/www/log. I’ve grown a habit of using the more detailed combined log style; if you’re fine with the default common format, you can leave the respective line out. Finally the configuration block defines a special rule for location / which means somebody accesses the virtual host directly (i.e. http://synth.local in this case). It will make the browser be redirected to the report index instead. Getting a special file (like e.g. lang___python37.log in the log directory) will not trigger the rule and thus still work. This is just a convenience thing and if you don’t like it leave it out.

All that’s missing now is starting up the webserver:

# service obhttpd start

You should now be able to point your browser at the the vhost’s name (if you made it resolve). Just using the machine’s IP address is also going to work in this case since it’s the default vhost. But better make it reachable using the configured name as we’re adding another vhost in a moment.

Synth web report for the latest package run

Authentication

But for now what about security? Let’s say you don’t want to share your report with the whole world. One easy means of protecting it is by using HTTP basic auth. OpenBSD HTTPd uses standard .htpasswd files. These can however use various cryptographic hashes for the passwords – whereas HTTPd only supports one: bcrypt.

The first time I tried to do authentication with OpenBSD HTTPd, it drove me completely nuts as I couldn’t get it working. Fortunately I own Michael W. Lucas’ excellent book “Httpd and Relayd Mastery”. After digging it out the index indicated that I might want to read page 28. I did, banged my head against the table and immediately got it working using that hashing algorithm. Don’t be like me, skip trying to use foreign tools you may be used to and just do it right in the first place. HTTPd comes with its own htpasswd binary. Use that.

In this example I’m adding a user called “synth”. Use whatever you prefer. Then give the password two times. This leaves you with a valid .htpasswd file that HTTPd could use – if it was allowed to access it! Let’s fix that problem:

# chown root:wheel /var/synth/www/.htpasswd
> # chmod 0640 /var/synth/www/.htpasswd

Having the authentication information in place, we only need to add another location block to the webserver’s vhost configuration. Put the following in there after the line that closes the previous location block:

    location "/Report/* {
        authenticate with "/.htpasswd"
    }

Note the htpasswd file’s location! It’s within the chroot (or it couldn’t be accessed by the webserver), but outside the webroot directory. So HTTPd could never accidentally serve it to somebody who knew that it was there and requested the file.

The only thing that remains is restarting the webserver. Next time you visit the report page, you’ll be asked to authenticate first.

# service obhttpd restart

Package repository

So far all of our packages have been created in a directory outside of the webserver’s chroot. If we want to make them available via HTTP, we need to use another path for them. Therefore we’re going to create a directory and reconfigure Synth again:

# mkdir -p /var/synth/www/packages
# synth configure

This time it’s setting B. Change it to /var/synth/www/packages and save. Now let’s build a package that draws in a couple of dependencies:

# synth just-build chocolate-doom

We can watch it now via the web reports while it’s building. Since it’s a new directory where no packages exist, yet, Synth is first going to build the package manager again. During this early stage no report is available, but once that’s finished the reports work.

While we’re rebuilding all packages due to the new package directory, Synth can take advantage of ccache as we haven’t modified it’s path. Wonder how much of a difference that actually makes? Building llvm10 on its own, one time using the cache and one time (for testing purposes) without it will show you the difference:

Duration: 00:13:32 (with ccache)
Duration: 02:09:37 (uncached)

Synth web report while it’s building

It gives us all the information that the curses UI holds – and more. The number of entries for completed packages can be changed. You can browse those page-wise back to the very first packages. It’s possible to use filters to e.g. just list skipped or failed packages. You can view / download (whatever your browser does) the log files for all those packages. And there’s even a search (which can be very handy if you’re building a large package set).

Report with only 10 entries per page

As long as packages are being built, the report page also shows the builder information and automatically refreshes the page every couple of seconds. Once it completes, it removes builder info (which would only waste space) and stops the polling. You can always come back later and inspect everything about the latest package run. The next one will overwrite the previous information, though.

Synth report search function

Now that we have a bunch of newly built packages, let’s see what that looks like:

# ls -1 /var/synth/packages/All
autoconf-2.69_3.txz
autoconf-wrapper-20131203.txz
automake-1.16.3.txz
binutils-2.33.1_4,1.txz
bison-3.7.5,1.txz
ca_root_nss-3.63.txz
ccache-3.7.1_1.txz
celt-0.11.3_3.txz
chocolate-doom-3.0.1.txz
cmake-3.19.6.txz
curl-7.76.0.txz
db5-5.3.28_7.txz
docbook-1.5.txz
docbook-sgml-4.5_1.txz
docbook-xml-5.0_3.txz
docbook-xsl-1.79.1_1,1.txz
doom-data-1.0_1.txz
evdev-proto-5.8.txz
expat-2.2.10.txz
flac-1.3.3_1.txz
[...]

Showing only ignored packages in the report (none in this case)

The packages are there. But what’s in the main directory?

# ls -l /var/synth/www/packages
total 18
drwxr-xr-x  2 root  wheel  150 Jun  7 23:57 All
drwxr-xr-x  2 root  wheel    3 Jun  7 23:21 Latest

This is not a valid pkg(8) repository. Which is no wonder since we used just-build. So we’re going to have Synth create an actual repository from these packages next:

Searching in the report after the build was completed

# synth rebuild-repository
# ls -l /var/synth/www/packages
total 117
drwxr-xr-x  2 root  wheel    150 Jun  7 23:57 All
drwxr-xr-x  2 root  wheel      3 Jun  7 23:21 Latest
-rw-r--r--  1 root  wheel    163 Jun  8 00:02 meta.conf
-rw-r--r--  1 root  wheel    236 Jun  8 00:02 meta.txz
-rw-r--r--  1 root  wheel  40824 Jun  8 00:02 packagesite.txz

Here we go, that’s all that pkg(8) needs. Synth should have automatically updated your repository configuration to use the new location. Have a look at /usr/local/etc/pkg/repos/00_synth.conf – the URL should point to the new directory.

Serving the repository

The next step is to make the repository available in the network, too. So edit /usr/local/etc/obhttpd.conf once more and add another “server” (i.e. vhost):

server pkg.local {
    listen on * port 80
    root "/packages"
    log style combined
    location * {
        directory auto index
    }
}

One service restart later you should be able to access the repository via a web browser from any machine in the same subnet (if you got your DNS right):

# service obhttpd restart

Looking at the package repository with a browser

This is already it, but let’s prove that it works, too. I’m adding the “pkg.local” name to the machine’s 127.0.0.1 definition in /etc/hosts, then change the URL in the Synth repo to fetch packages via HTTP:

  url      : http://pkg.local,

I’ve also created a FreeBSD.conf to disable the official repository. Let’s stop the webserver for a second and then try to update the package DB:

# service obhttpd stop
# pkg update
Updating Synth repository catalogue...
pkg: Repository Synth has a wrong packagesite, need to re-create database
pkg: http://pkg.local/meta.txz: Connection refused
Unable to update repository Synth
Error updating repositories!

Ok, so there’s no other repository configured anymore and this one is not accessed via the filesystem. So we’re going to start the webserver once more (give it a sec) and then try again:

# service obhttpd start
# pkg update
Updating Synth repository catalogue...
Fetching meta.conf: 100%    163 B  0.2kB/s    00:01
Fetching packagesite.txz: 100%   40 KiB  40.8kB/s   00:01
Processing entries: 100%
Synth repository update completed. 148 packages processed.
All repositories are up to date.

Great! So now we can install DooM on this box or on any other machine running FreeBSD 13.0 which can reach it over the network.

Package sets

So far we’ve only either built all packages for what was already installed on the machine or for single ports that we selected at the command line. But now that we can serve packages over the network, it’s rather tempting to use a powerful build machine to build packages for various other FreeBSD machines, isn’t it? Let’s assume that you’re going to share packages with a KDE lover.

First we should prepare a list of packages that we need, starting with what is installed on our machine.

# pkg info | wc -l
345

Wow, that’s already quite some packages for such a pretty naked system! But we don’t need to consider them all as most of them are just dependencies. Let’s ask pkg(8) for the origin of all packages we explicitly installed (i.e. which were not recorded as automatically installed):

# pkg query -e %a=0 %o > /var/synth/pkglist
# cat /var/synth/pkglist
x11-wm/awesome
devel/ccache
graphics/drm-kmod
devel/git
www/obhttpd
ports-mgmt/pkg
ports-mgmt/portmaster
x11/sakura
x11/setxkbmap
security/sudo
ports-mgmt/synth
sysutils/tmux
x11/xfce4-screenshooter-plugin
x11/xorg-minimal

That’s better! But we certainly don’t need portmaster anymore, so we can take it off the list (and deinstall it actually). Let’s add www/firefox and x11/kde5 for our pal (and sort the list since it’s a bit ugly right now).

Once that’s done, we should be able to do a simple:

# synth build /var/synth/pkglist
Error: port origin 'devel/git' not recognized.
Perhaps you intended one of the following flavors?
   - devel/git@default
   - devel/git@gui
   - devel/git@lite
   - devel/git@svn
   - devel/git@tiny

Oh yes, right! We need to edit our list and specify the flavor to build! I’m going with the lite variant here, so the git line needs to be changed to this:

devel/git@lite

Then we can try again – and yes, it starts building after calculating the required dependencies.

Logs

Whoopsie! After about an hour stupid me removed the network cable for a moment. This has caused a couple of build failures (see screenshot). The report will display the phase that the build failed. In this case it’s the fetch phase (and we don’t have to look for the reason as we already know it). Sometimes a distfile mirror is temporary down or the distfile has been removed. In that case you will have to manually get the files and put them into the distfiles directory. Skipped ports also display the reason, i.e. which dependency failed previously.

Failed and skipped ports due to a connection problem

I better re-attach that cable right away and start the building over… Many hours later it has finished. But what’s this? Rust has failed again (and this time it wasn’t me)! And it failed at the stage phase. When this happens it’s usually because of a broken port got committed. Update your ports tree and hope that it has been fixed in the meantime. This is not the reason in our case, however.

Another phase, another failure!

But how do we find out what actually happened? Well, by looking at the logs, of course. Here’s the last 15 lines of the relevant log:

        finished in 183.141 seconds
  < Docs { host: TargetSelection { triple: "x86_64-unknown-freebsd", file: None } }
Install docs stage2 (Some(TargetSelection { triple: "x86_64-unknown-freebsd", file: None }))
running: "sh" "/construction/xports/lang/rust/work/rustc-1.51.0-src/build/tmp/tarball/rust-docs/x86_64-unknown-freebsd/rust-docs-1.51.0-x86_64-unknown-freebsd/install.sh" "--prefix=/construction/xports/lang/rust/work/stage/usr/local" "--sysconfdir=/construction/xports/lang/rust/work/stage/usr/local/etc" "--datadir=/construction/xports/lang/rust/work/stage/usr/local/share" "--docdir=/construction/xports/lang/rust/work/stage/usr/local/share/doc/rust" "--bindir=/construction/xports/lang/rust/work/stage/usr/local/bin" "--libdir=/construction/xports/lang/rust/work/stage/usr/local/lib" "--mandir=/construction/xports/lang/rust/work/stage/usr/local/share/man" "--disable-ldconfig"
install: creating uninstall script at /construction/xports/lang/rust/work/stage/usr/local/lib/rustlib/uninstall.sh
install: installing component 'rust-docs'
###  Watchdog killed runaway process!  (no activity for 78 minutes)  ###



--------------------------------------------------
--  Termination
--------------------------------------------------
Finished: Wednesday, 9 JUN 2021 at 00:04:47 UTC
Duration: 04:27:03

Ha! The build process was killed by the watchdog! Bad doggy? It does happen that the process would eventually have finished. Not this time. We have to dig a little deeper. In /var/log/messages of the build machine I can find the messages kernel: swap_pager: out of swap space and kernel: swp_pager_getswapspace(4): failed. This machine has 24 GB of RAM and 8 GB of swap space configured. And by building 6 huge ports concurrently, it exceeded these resources! Keep in mind that package building can be quite demanding, especially if you use tmpfs (which you should if you can).

So, there we are. We’ve configured our build server for web reports and serving the repository. We’ve looked at building package sets and covered a few examples of what can go wrong. And that’s it for today.

What’s next?

The last article about Synth will cover make.conf, signing repositories and using cron for automated builds. We’ll also take a brief look at profiles.

FreeBSD package building pt. 3: Intermediate Synth

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/freebsd_package_building_pt3.gmi

In this article we’ll continue exploring Synth. After covering a general introduction to package building as well as Synth’s basic operation in the previous articles, we’re going to look at the program’s configuration and using it for updating the system.

The first thing to do here is to get rid of the old ports tree and replace it with a newer one so that some updates become available:

# zfs destroy zroot/usr/ports
# zfs create zroot/usr/ports
# git clone --depth 1 -b 2012Q2 https://git.freebsd.org/ports.git /usr/ports
# mkdir /usr/ports/distfiles
# synth status

Synth regenerates the flavor index again, then gives the output. This time it will not just show all packages as “new” but most are either updates or rebuilds.

Synth status with new ports tree

Configuration

Let’s look at how you can change Synth’s behavior next:

# synth configure

Synth’s configuration is pretty straight-forward. Options A to G configure the various directories that the program uses. Should you change anything here? If you know what you are doing and have special needs: Maybe. If you actually ask the question whether to change any such directory, the answer is no. Option H (Compiler cache) is a special one. We’ll get to that in a minute.

With I you set the number of builders that Synth uses. J means how many threads each builder may use; think make -j $SOMENUMBER. When compiling a single piece of software, it’s usually recommended to set the number of jobs equal to the machines core count + 1. Take a look at the screenshot above: Doing the math, we’re configuring for 24 cores here – on a system that has 8 (with Hyper Threading).

Synth’s configuration menu

Why does Synth choose to over-provision the resources so much? The reason is simple: It’s only over-provisioned when more than three builders are active at the same time. Often enough not all builders will be at the build stage (where the compiling happens) at the same time. Most other stages are much lighter on CPU – which would mean that you’re wasting available resources (and thus prolong the total build time). Also in the previous post you’ve seen that LLVM and Rust took hours to build (all four remaining builders were idle most of the time!). If the cap would have been lower, build times would have increased even more.

So what’s the best setting for builder count and max jobs? There’s no single answer to that. It depends on both your machine and on the set of ports that you build. Play with both values a bit and get a feeling what works best for you. Or leave it at the default that Synth calculated for your system which will probably work well enough.

Options K and L are real speed boosters. They control whether the base system for the builder (“localbase”) and the directory where the program is built are using tmpfs or not. Tmpfs is a memory-backed filesystem. If you disable one or both options, compilation times increase a lot because all the data that needs to be copied on setting up a builder and compiling software will be written to your disk. For an extreme example: On one of my machines, building and testing a port increased from slightly over 30 seconds with tmpfs to over 4 minutes (!) without it. Yes, that machine uses an HDD and it was occupied with other things besides building packages. But there is more than a negligible impact if you disable tmpfs.

So when to do it in the first place? If you’re on a machine with little RAM you might have to do this. Some ports like LLMV or Firefox require lots of RAM to build. If your system starts swapping heavily, disable tmpfs. Ideally build those ports separately and leave tmpfs on for all the rest.

Then there’s option M which toggles the fancy colored text UI on or off. If you turn it off, you’ll get a simple monochrome log-like output of which builder started or finished building which port. Every now and then the info that’s in the top bar of the UI (e.g. number of packages completed, number of packages remaining, skips, etc) gets printed.

Finally we have option N which toggles using pre-built packages on or off. It’s off by default which means that Synth will build everything required for the selected package set. If you enable this option and have e.g. the official FreeBSD repository enabled as well, it will try to fetch suitable packages from there that can be used as the buildtime or runtime dependencies of the packages that still need to be built. If you’re mixing official and custom packages this could be a huge time saver. I admit that I have never used this option.

And what’s this profile thing? Well, ok. The configuration we’ve been looking at is for the default profile. You can create additional ones if you want to. And that’s where the directories that I told you to ignore come into play. You could for example create a profile to use a different ports tree (e.g. the quaterly branch) and switch between the profiles to build two different packages sets on one machine. While Synth can do this, that is the point where I’d advise you to try out Poudriere instead. The main benefit of Synth over Poudriere is ease of use and when you’re trying to do clearly advanced things with Synth you might as well go for the officially supported FreeBSD package builder instead.

Compiler cache

Let’s disable the curses UI just to see what Synth looks like without it and save the config change. If you plan to build packages regularly on your system, you will definitely want to set up the compiler cache. To be able to use it, we first need another package installed, though: ccache. We’re going to build it and then install it manually this time:

# synth just-build devel/ccache
# pkg add /var/synth/live_packages/All/ccache-3.7.1_1.txz

Then we’re going to create a directory for it and go back to Synth’s configuration menu:

# mkdir -p /var/tmp/ccache/synth
# synth configure

Now change H to point to the newly created directory and save. Synth will use ccache now. But what does it do? Ccache caches the results from compilation processes. It also detects if the same compilation is happening again and can provide the cached result instead of actually compiling once again. This means that the first time you compile it doesn’t make a difference but after that the duration of building packages will drop significantly. The only cost of this is that the cached results take up a bit of space. Keep ccache disabled if drive space is your primary concern. In all other cases definitely turn it on!

Ccache directory and config after the system upgrade

Updating the system

Next is using Synth to update the system.

# synth upgrade-system

After determining which packages need to be built / rebuilt, Synth will start doing so. We’ve turned off the text UI, so now we get only the pretty simplistic output from standard text mode.

Package building in pure text mode

It provides you with the most important information about builders: Which builder started / finished building which package when. You don’t get the nice additional info about which state it’s in and how long it has been busy building so far. As mentioned above, Synth will print a status line every couple of minutes which holds the most important information. But that’s all.

Status lines amidst the builder output

What happens if something goes wrong? I simulated that by simply killing one of the processes associated with the rust builder. Synth reports failure to build rust and prints a status line that shows 17 package skips. In contrast to the curses UI it does not tell you explicitly which ones were skipped, though!

Simulated package build failure

When Synth is done building packages, it displays the tally as usual and does some repository cleanup by removing the old packages. Then it rebuilds the repository.

Tally displayed after completion and repository cleanup

Since we asked Synth to upgrade the system, it invokes pkg(8) to do its thing once the repository rebuild is complete.

Package repository rebuilt, starting system upgrade

And here’s why I strongly prefer prepare-system over upgrade-system: The upgrade is initiated whether there were failed packages or not. And since pkg(8) knows no mercy on currently installed programs when they block upgrades, it will happily remove them by default! To be safe it makes sense to always review what pkg(8) would do before actually letting it do it. Yes, it’s an additional step. Yes, most of the time you’ll be fine with letting Synth handle things. But it might also bite you. You’ve been warned.

System upgrade in progress – it may remove packages!

Every now and then you may want to run synth purge-distfiles (unless you have unlimited storage capacity, of course). It will make the tool scan the distinfo files of all ports and then look for distfile archives of obsolete program versions to remove.

Cleaning up old distfiles with purge-distfiles

There might not be a large gain in our example case, but things do add up. I’ve had Synth reclaim multiple GB on a desktop machine that I regularly upgraded by building custom packages. And that’s definitely worth it.

What’s next?

The next article will cover some leftover topics like logs, web report and repository sharing.

FreeBSD package building pt. 2: Basic Synth

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/freebsd_package_building_pt2.gmi

The previous article was an introduction into package building on FreeBSD in general. It also included references to various other articles about package management and working with ports. Familiarity with those topics is assumed here. If you don’t feel confident, have a look there and do a bit of reading first. Or just go ahead and do some research if you hit something that you have trouble understanding – package building is not a beginner’s topic but it ain’t rocket science, either.

In the following section I’m following a “naive approach” of just installing and exploring Synth on a typical desktop system. Afterwards I’m switching to the test system that was described in the previous post so that I can show updating, too.

Synth basics

So let’s get started with Synth now, shall we? The first thing to do is of course installing it:

# pkg install synth

If you want to or need to build it from source instead, the origin is ports-mgmt/synth. It has one compile-time option: Enabling or disabling the watchdog feature. By default it is enabled which means that Synth is supervising the build processes and if one looks like it has stalled, the watchdog will kill it. When you are building on very slow hardware (why are you doing this in the first place?) it may be an anti-feature that causes very complex ports (e.g. Firefox) to be killed even though it would otherwise eventually finish. In that case you can disable the watchdog option and rebuild Synth. Or you go to the junkyard and get some more capable hardware!

Synth port options

As with every new piece of software, it’s probably a good idea to get an overview of what it can do:

% synth help
Summary of command line options - see synth.1 man page for more details
===============================================================================
synth status              Dry-run: Shows what 'upgrade-system' would build
synth configure           Brings up interactive configuration menu
synth upgrade-system      Incremental rebuild of installed packages on system.
                          Afterwards, the local repository is rebuilt and the
                          system packages are automatically upgraded.
synth prepare-system      Like 'upgrade-system' but ends when repo is rebuilt
synth rebuild-repository  Rebuilds local Synth repository on command
synth purge-distfiles     Deletes obsolete source distribution files
synth status-everything   Dry-run: Shows what 'everything' would build
synth everything          Builds entire ports tree and rebuilds repository
synth version             Displays version, description and usage summary
synth help                Displays this screen
synth status [ports]      Dry-run: Shows what will be rebuilt with given list
synth build [ports]       Incrementally build ports based on given list, but
                          asks before updating repository and system
synth just-build [ports]  Like 'build', but skips post-build questions
synth install [ports]     Like 'build', but upgrades system without asking
synth force [ports]       Like 'build', but deletes existing packages first

synth test [ports]        Just builds with DEVELOPER=yes; pre-deletes pkgs

[ports] is a space-delimited list of origins, e.g. editors/joe editors/emacs.
It may also be a path to a file containing one origin per line.

That’s a nice list and even better: The commands are explained in a way that mere mortals can understand. Basically it can compare installed program versions to those in ports for you with synth status. You can configure synth with synth configure (we’ll look at this later, the default config is fine for now). And it can bulk-build packages and use those to upgrade the system for you with synth upgrade-system. And those are the important ones that you’ll need to begin with.

There’s also synth prepare-system in case you want Synth to only build the packages (maybe you want to do the upgrade manually later). With synth rebuild-repository you can make the tool, well, rebuild the repository from the currently built packages. Since it does this after finishing package builds, you will need it only if you cancelled building somewhere in the middle and want it to rebuild the repo anyway. The synth purge-distfiles command is useful if you use Synth for a while. It will scan for no longer needed distfiles (for obsolete program versions) and potentially free quite a bit of space for you.

And that’s it. The synth status-everything and synth everything commands are only useful if you want to build packages for the entire FreeBSD ports collection. But that’s certainly not basic usage! The various options that act on a single port or a list of ports is also advanced usage. They’ll come in handy if you plan on developing ports, build packages for other machines to use or have special needs. If you plan to keep your machine’s installed packages consistent for production use, know that these commands exist but stay away.

Package status

So much for some theory and on to actually doing something. Let’s ask Synth about the status of our packages:

# synth status
It seems that a blank PORTSDIR is defined in /etc/make.conf
Configuration failed to load.

Alright, this message can be a little confusing as by default there is no such file in FreeBSD! What Synth actually wants to tell you is that it cannot find a ports tree on the system and thus cannot do anything. So let’s get the ports in place right now:

# portsnap auto

With the ports tree available on our system we can try again:

# synth status
Configuration invalid: [C] Distfiles directory: /usr/ports/distfiles
Rather than manually creating a directory at this location, consider
using a location outside of the ports tree. Don't forget to set
'DISTDIR' to this new location in /etc/make.conf though.

This time the problem is pretty straight-forward: The distfiles directory does not exist. Synth gives us some advice in case we want to use a custom location, however I prefer the standard one.

# mkdir /usr/ports/distfiles
# synth status
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
 progress: 3.32%

Depending on your build machine this can take anything from a couple of minutes to a quarter of an hour or so. Before FreeBSD introduced port flavors, it was a matter of seconds, but for a few years now Synth has to do some additional work up-front. When it’s done it will print something like this:

Querying system about current package installations.
> Installed package ignored, devel/py-evdev package unmatched
> Installed package ignored, devel/py-pyudev package unmatched
> Installed package ignored, devel/py-six package unmatched
> Stand by, comparing installed packages against the ports tree.
> Stand by, building pkg(8) first ... done!
> These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
>   N => print/indexinfo
>   N => devel/gettext-runtime
> [...]

For the first run, Synth has to build all of the packages. Therefore all are marked new. But what’s that “Installed package ignored” thing? That usually happens if a package is installed on the system that was built from a port that does no longer exist in the new tree. Sometimes it’s also weirdness that can happen with flavored ports. Disregard this if it’s only affecting packages that you don’t want to use directly. The correct ones will be pulled in as dependencies anyway. Now let’s build our first package set:

# synth prepare-system

After gathering some information, Synth will start building the packages that are currently installed on this machine (which for simplicity’s sake resemble those of the test system). Let’s have a look:

Synth started building packages

By default, Synth shows a nice curses-based text UI that displays a lot of information (see screenshot above). It shows the total count of packages to build, the number of packages that remain to be built and the count of already built packages. In case a port does not successfully build on this system (update ports and try again – if the problem persists either file a bug report or contact the maintainer), Synth displays that, too. Ignored packages are those that don’t work on your particular system; maybe the application is known to not build on the version of FreeBSD you are using or not on your architecture (if for example you’re building on ARM). Skipped count goes up if a failed port was a dependency for others. Obviously Synth cannot build them.

Synth also displays the current system load: A load of e.g. 2.0 means that two cores of your CPU are completely busy. If your CPU supports Hyper-threading that basically doubles the available core count. A load higher than 8.0 on a 4 core system with HT means that the system has currently more work than it’s able to fulfill concurrently. It’s not ideal but it’s not something to worry too much about, either. Watch swap usage. A little bit of swapping for large ports is not much of a problem. But if your system swaps a lot that will slow down the package building a lot. Should you manage to run out of swap, package builds will fail. You can adjust Synth’s configuration if you’re unhappy with either load or swap usage. But we’ll get to that.

Synth after about 20 minutes

There’s also the number of packages built per hour and the impulse as well as the elapsed time. Initially packages per hour and impulse are identical because the two are similar. However the former is the average number of packages built over the whole build time while the latter is the number of packages built within the last 500 seconds.

But that’s only the top bar. The next part of the UI is for the builders. Builders are clean build environments (“chroot jails”) that are created before building a package and then teared down afterwards. The next package gets a new fresh environment. The number of builders means how many packages can be building concurrently. On the screenshot you can see 6 builders which are color-coded to be easier to distinguish. This may look different on your machine and here’s why: Synth tries to guess reasonable defaults for your machine. If you have an old dual-core PC, it will use less than the six builders it deemed right for the quad-core i7 that I used as my test machine here. Expect to see a much higher number of builders on modern servers with a higher core count.

Idle builders after 5 hours

For each builder you see how long it has been working already, which phase it is currently in (remember the various build targets of the ports infrastructure?) which port the builder is occupied with and how many lines of log output the builder has has produced so far. Especially the last information is not going to help you a lot when you begin building packages. After a while you’ll know roughly how many lines of output some of the big ports like LLVM produce and can judge if it’s going to take another two hours to finish or more like half an hour. (And then a new version of LLVM comes out which takes even longer to build so that your previous idea of “how many lines” is no longer valid. That’s how things go.)

And finally there’s the rest of the screen made up of a list of recently finished packages. If you take a look at the second screenshot, you’ll see some ports where the origin ends with @py37. Here the builder is busy building a flavored port – and it’s building for Python 3.7. What’s @native you ask? Well, Python ports are a typical example of flavored ports but they are not the only ones. The binutils port for example is able to be built as part of a native toolchain or a cross toolchain in case you want to e.g. cross compile packages for riscv on your much more powerful amd64 machine.

First builder has shut down

What’s the deal with idle builders like those on screenshot 4? Idle builders are the ones that Synth has already prepared for another package to build in but has not been able to use so far. Take a look at screenshot 4: There’s 53 more packages to build but only two builders are occupied with work while four are idle. Can you guess why? The two ports currently building are LLVM10 and Rust. And the “problem” is that all other 51 packages that are still on the list depend on either LLVM or Rust directly or indirectly! So while there’s more work for the builders, Synth has to wait for the dependencies to finish building before it can start building the remaining packages.

At some point no further builders will be required. In that case Synth shuts them down (see screenshot 5).

Last package build starting after about 6 hours

When Synth has finished building all the packages requested, it will present you with the final statistics. Then it cleans up the package directory, removing obsolete old packages. Eventually it will build a valid pkg(8) repository for you – and then it’s finally done.

Synth has completed its task

Should you ever want to quit a package run early, try to avoid CTRL-C. If you press CTRL-Q instead, Synth will shutdown gracefully. This means that now new builders will be started and the tool exits properly once those that are already running complete.

Pkg(8) repositories

I’m covering the case where you want to use your own package repository instead of the official FreeBSD one. If you want to use both, make sure you read what I pointed my readers to in the previous article. Then configure your new repository as I do here but simply don’t disable the official repo.

The standard FreeBSD repository is configured in /etc/pkg/FreeBSD.conf. Do not make changes there, though! This file belongs to the base system and upgrades may overwrite it. If you want to change the settings, create /usr/local/etc/pkg/repos/FreeBSD.conf (and the missing directories). The latter file will override the former. In case you just want to disable the official package repository, simply put this single line into the file:

FreeBSD: { enabled: no }

Synth should automatically generate a config file there if you use it to upgrade the system. The file is called /usr/local/etc/pkg/repos/00_synth.conf and has the following content:

# Automatically generated.

Synth: {
  url      : file:///var/synth/live_packages,
  priority : 0,
  enabled  : yes,
}

Now you only need to update the repository information:

# pkg update

And that’s it. Pkg(8) will now happily use your local package repository.

What’s next?

The next article will feature Synth configuration, upgrading and advanced usage.

FreeBSD package building pt. 1: Introduction and test system

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/freebsd_package_building_pt1.gmi

In 2017 I started to post a series of well-received articles that was meant to culminate in a tutorial on building packages on FreeBSD. I wrote about the history of package managers, about how to use FreeBSD’s pkg(8) package manager (part 1 and part 2) as well as an introduction to working with ports (part 1 and part 2).

Then I had to stop for technical reasons (after a major change to the ports infrastructure, the tool that I wanted to write about had not been updated to work with the new ports tree, yet!). In 2019 I eventually found the time to publish two more articles that were originally meant to come right after the 2017 posts. They covered using classic tools for ports management (part 1 and part 2).

They were meant to be the stepping stone to what I actually wanted to cover: Package building! The second article ended with:

I’ve planned for two more articles that will cover building from ports the modern way(tm) – and I hope that it will not take me another two years before I come to it…

That was… Yikes! In fall 2019… So I’d better hurry up now. I had hinted that using ports directly was not the recommended thing to do anymore and that while you should use packages unless you need ports, you really should use packages even in the latter case! And that’s what we’re going to do in the next few articles.

Why roll your own packages?

There are valid reasons to not use FreeBSD’s official packages:

  • Most frequently you need build-time options configured differently.
  • Or your organization uses a different default version of a scripting language (Perl, Python, Ruby, …) that what FreeBSD chose at the time being.
  • Maybe you’re running a customized version of FreeBSD that’s not officially supported.
  • Perhaps you need programs where the license forbids binary redistribution.
  • Or you use custom ports for some programs that of course cannot be available in official packages then.

Things like that. People choose ports when they need more control over their applications. There are good reasons to avoid using ports the traditional way, too, however:

  • It takes time and resources to build everything from source; especially if you do this on multiple machines, it’s a waste.
  • It makes updates much more complicated (see the second 2019 post mentioned above).
  • It clutters your system with build-time dependencies that need to be installed first so that the actual programs can be built.
  • Depending on what other software your machine has installed, the resulting programs might differ from that of other machines even if they built the same port!

While the former two points are mostly relevant if you manage multiple machines, I’d recommend rolling your own packages even for the single FreeBSD workstation that you use – if the official packages don’t suit you. Let me state this again: Try to go with the packages provided by the FreeBSD project first. Build your own packages if you have to (educating yourself is a completely valid reason to do it anyway, though).

Package builders

When you decide to roll your own packages, you have two options: Synth, the much easier and nicer package builder and Poudriere, the advanced build tool that FreeBSD uses and provides official documentation for.

Which one should you choose? I’m going to show how to work with both so you can compare them and make an informed decision. If you’re just getting started, you may want to give Synth a try. It is also a good choice when you use DragonFly BSD, too: The dsynth tool that they have in base was inspired by Synth (and if it wasn’t written in Ada they certainly would just have imported it instead of creating a re-implementation in C). You should also know that the Synth project is in maintenance mode. Its author still fixes bugs and takes pull requests on GitHub, but it’s feature-complete.

The main advantage of Synth is that it’s dead simple to setup and use, too, whereas Poudriere is a somewhat complex beast. Synth also shines when you want to use it to keep one machine up to date with packages as it can do those updates for you. Poudriere on the other hand allows you to do things like maintaining package repositories for multiple versions of FreeBSD as well as multiple architectures from one build machine. If you need that, forget Synth.

Ports and Git

One major change that was made in FreeBSD since the previous article was published is that the project migrated to using Git repositories. FreeBSD development started on CVS but in 2008 the src repository was successfully migrated to using Subversion. In mid 2012 docs and ports also made the switch. Subversion has been used ever since until December 2020 when doc and src transitioned to Git. After some remaining issues were solved, ports also migrated to Git in April 2021. While src changes get pushed back to Subversion for FreeBSD’s branches of 11 and 12, when it comes to ports, an era has ended.

Get rid of that habit of visiting svnweb.freebsd.org and start getting used to working with cgit.freebsd.org instead.

If you are unsure of your Git skills, you may want to at least skim over the Git-primer in FreeBSD’s documentation.

At least get familiar with the concepts. You should for example know that Git is able to do things like a shallow clone; looking things up when you need them is no problem but not being aware of them at all is.

While both Subversion and Git are used for version control and both use repositories, they are fundamentally different. Subversion is the best known version control system of the second generation (so-called (centralized) “networked VCS”). Git is the most popular one of the third generation (“decentralized VCS”).

If you use Subversion, there’s a central repository somewhere and you checkout a certain revision of the versioned files from there. The files that were checked out form your working directory. In Git you clone a remote repository and thus receive all the data needed to get a local copy of the whole repo. The files for the working directory are checked out from the local copy. The major difference is: You have the whole history available locally.

In case you have no need for all that data, you do a shallow clone instead of a regular full clone. To give you an example: If you do a shallow clone of the current ports tree today, the result is about 840 MB in /usr/ports – of which 85 MB in size is the Git repository. A full clone is about 1.7 GB in size of which about 920 MB is for the repo! So if you don’t need the history, save some space on your drive and save some donated bandwidth of the FreeBSD project.

Considerations before you start

While you can certainly start rolling your own packages on a fresh system, it’s also fine to begin doing so on a system that has used the standard FreeBSD ports or packages so far. There’s nothing wrong with that actually. Taking the opposite way and going back from custom packages to the official ones is also possible, of course. That latter case requires some more planning, though. Think about why you started building your own packages in the first place, find out if the standard packages fit your current needs and think about the consequences. If you’re uncertain, you may want to start over with regular packages. If you want to make the switch back, it’s best to re-install all the packages (use the flag -f with pkg upgrade). If you’re using ZFS in a way that supports Boot Environments, create one first.

In fact you can even choose to use both official and custom packages on the same machine: Instead of building the full set of packages that you need, you build just the packages yourself that you need to customize and use the packages from the standard FreeBSD package repositories for the rest. This works by configuring your package repository as an additional one for pkg(8) and giving it a higher priority. If you consider doing this and using multiple package repositories, be sure to

% man 5 pkg.conf

first. It doesn’t hurt to skim over the whole manpage, but make sure that you at least read the section REPOSITORY CONFIGURATION. I used that approach on a weaker machine in the past, but wouldn’t generally recommend it. If your hardware is recent enough, you can compile everything yourself. Otherwise it might make more sense to build on a somewhat beefy system and distribute the packages to your other machine(s). But that’s only me. Maybe mixing packages is the right solution for your use case.

Building a test system

The rest of this post is about building a test system to simulate an environment with pre-installed packages. I do this so that I can show off a few things regarding updates in the next post. If you want to follow along, make sure that you have Git installed and that your directory or dataset /usr/ports as well as /usr/src is empty. I’m assuming that you’re using FreeBSD 13.0.

The first thing to do is getting the ports tree in place. I’m doing a shallow clone of the 2021 first quarter branch (so I can update later). Then I clone operating system source repository as well:

# git clone --depth 1 -b 2021Q1 https://git.freebsd.org/ports.git /usr/ports
# git clone --depth 1 -b releng/13.0 https://git.freebsd.org/src.git /usr/src

Now we don’t need the packages anymore and forcefully remove them all (including pkg). The reason for this is that we want to build the older versions of the old ports tree that we just cloned.

# pkg delete -af

For convenience we’re going to build all the required programs using portmaster as discussed in a previous article (see the beginning of this post). Of course we need to build and install it first:

# make -C /usr/ports/ports-mgmt/portmaster install clean

Alright. Now we’re building some leaf ports to simulate a very simple development system. These ports draw in a number of dependencies so that we end up with about 350 packages (which is still a very low count for a desktop system). I’m building tmux and sudo simply because I always need to have them available and git just because we needed it before anyway. The rest is the graphics drivers, a simple subset of X11 + the command to set a different keyboard layout as well as the awesome window manager and the simple but nice GTK+ terminal emulator called sakura. Since I’ll need to take screenshots for the upcoming posts, I figured that I might include an application for that, too.

# portmaster -dG sysutils/tmux security/sudo devel/git graphics/drm-kmod x11/xorg-minimal x11/setxkbmap x11-wm/awesome x11/sakura x11/xfce4-screenshooter-plugin

And that’s it for today.

What’s next?

Next station is taking a look at building packages and updating the system using Synth.

FreeBSD router take 2 (pt. 4): Demoting my ISP’s router

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/bsd_router_take_2_pt4.gmi

Since I built my first OPNsense-based router, it had been a secondary router only. Its “WAN” port was connected to my ISP’s modem/router box which dealt with establishing the actual Internet connection and acting as the gateway and DHCP server for my OPNsense. In other words: It has only ever been a second line of defense for my LAN network behind it.

Also since I started building said first router of mine I had the goal of eventually replacing my ISP’s router with it instead of just adding a secondary device. In retrospect it still was the right choice to start with a first step like I did. It allowed me to play with OPNsense and get familiar with it while I did not fully depend on it working right all the time. It has been fun and the fact that OPNsense never broke for me when updating was a reassuring experience.

Promoting OPNsense to be my primary router

The following image shows what my network looked like up until a short while ago (red = my ISP’s property, blue = OPNsense, green = transparent networking devices):

Diagram of my network before the change

As you can see, my ISP’s modem/router was the device directly connected to my ISP’s line, forming my primary network behind it (LAN 1). Occasionally I’d plug a laptop or something into that directly. The APU running OPNsense was a secondary router behind which the network for my regular devices began (LAN 2). It meant that I could better tune packet filtering rules than the primary router allowed me. I could do proper QoS and other things. But it only was an additional device and didn’t really obsolete the one that I actually wanted to get rid of.

I did a bit of research and finally in early April I went ahead and made the switch. For my network OPNsense is in charge now! But I didn’t actually get rid of my ISP’s old box just yet. Why? Well… because of IP telephony. This is a topic of its own and I hadn’t been inclined to doing too many things at once. So I decided to try out if my phone would still work if I demoted my ISP’s router to be the secondary router and would let it manage telephony. I just set the device to operate in client mode, connected the IP phone, tried to call my parents – and was very relieved to find that it just worked!

So this is the new network diagram:

My network after the change

The APU is not technically capable of connecting with my ISP; I needed a DSL modem for that. The model that I chose operates in bridge mode so that it’s network-transparent but let’s OPNsense establish the connection. Directly connected to that additional device is the APU which now also handles a second network segment via its OPT1 interface: My old router is connected to that.

At the end of the day I have one more device involved but packets originating from my main computers no longer have to go through two routers. VoIP packets have to now, but at least the primary router is the box that I control, so that’s an improvement as well.

Choosing a DSL modem

After my research, I settled on the Zyxel VMG1312-B30A which is marketed as a “Wireless N VDSL2 4-port gateway with USB”. It’s an older device from 2012, but it’s still sold. While the specs don’t look very impressive today, I don’t care about which wireless standard it supports and such. I got it for another feature that it offers: Bridge mode.

Zyxel VMG1312-B30A DSL modem top view

If I were to simply replace my ISP’s device with this one, it’d even be a downgrade – the Zyxel makes an even worse router than what I had. But operating in bridged mode it simply terminates the DSL circuit of the telephone line and communicates using the DMT (Discrete Multitone Modulation) protocol with the ISP’s DSLAM (Digital Subscriber Line Access Multiplexer).

Zyxel VMG1312-B30A DSL back and bottom top view

Before settling on a modem, do some research on which technology is being used for DSL in your country! For my Zyxel device there are two separate variants of the same model: One for the so-called Annex-A and one for Annex-B. The former specifies “DSL over POTS” (= Plain Old Telephone Service) while the latter is for “DSL over ISDN”. Both devices are physically different, so be sure to get the right one (in Germany for example it’s Annex-B)! Annex-A uses the smaller RJ-11 jacks to connect to the ISP line while Annex-B uses the standard RJ-45 jacks that are also used for ethernet cables.

Taking a first look at the modem

By default the modem operates in router mode and has the IP 192.168.1.1/24 assigned. Configuration is accessible via a web UI on ports 80 and 443. The user is admin and the password 1234. Configure a workstation a static IP in the same subnet and connect it to the device, then login.

The modem has a standard overview page called “connection status” and four more sections that offer a menu each. First one is Network Setting.

“Network Setting” menu

The most important pages in that menu are Broadband, Wireless, Home Networking and Power Management.

Menu number two is called Security.

“Security” menu

This time there are two interesting pages: Firewall and MAC filter.

The third menu is “System Monitor”.

“System Monitor” menu

There you will find logs, the ARP table, the routing table and so on if you need it.

Finally there’s the “Maintenance” section.

“Maintenance” menu

The most important pages here are User Account, Remote MGMT, Firmware Upgrade and Configuration.

You should probably start by updating the firmware to the latest available version, but I’ll go through some of the pages here in order.

Modem configuration

First go to Network Setting -> Broadband. Set the device to operate in Bridge mode. Depending on what ISP you are using, you might need to have to set a VLAN tag for the connection to work. In my case choosing VLAN 7 is required. You might need to do some research or try out some possibilities.

Make the modem work in bridge mode

Then go to Network Setting -> Wireless. Do yourself a favor and just disable it. It will save you some power and offer additional security. If you need to update the firmware again or make another change, just physically connect a machine to it.

Turning off wireless access

Next is Network Setting -> Home Network. Turn off the DHCP server there. And if you’re paranoid, assign it a different IP – preferably in a different private address range.

Disabling the DHCP server

Lastly for the first section go to Network Setting -> Power Management. Here you can turn off everything that you don’t need. I only left the WAN port as well as one LAN port active and chose to unpower the rest.

Unpowering the LED and most ports

Next is Security -> Firewall. Since we’re not using Router mode, the firewall doesn’t make any sense. Off it goes.

Switching off the Firewall

For a bit of extra paranoia go to Security -> MAC Filter. Here you can choose to allow access to the modem only from certain NICs. If you consider doing this, make sure that you understand the consequences. Allow a minimal of two MAC addresses to not lose access if the respective NIC / machine should ever get damaged. If you only configure one, make sure to at least write it down and deposit it somewhere safe in case you need to spoof it. Otherwise you’ll have to factory-reset the modem when you managed to lock yourself out.

MAC Filter interface

Definitely go to Maintenance -> User Account and change the default password to something stronger.

User Account settings

Pay Maintenance -> Remote MGMT a visit. Turn off everything that you don’t need. You definitely don’t want Telnet, FTP or plain HTTP. Chances are that you don’t want SNMP either (if you do want to have it you know why). Disable ping if that makes you more feel better. And when it comes to SSH, here’s the reason I turned it off:

Unable to negotiate with [IP ADDRESS] port 22: no matching exchange method found. Their offer: diffie-hellman-group-sha1

This means that they ship a version of OpenSSH from 2015 or older (and probably never updated it since 2012 – if they even used the most current one back then). You can make your client talk to it anyway, but for me there’s generally no need for it.

Remote MGMT choices

Definitely go to Maintenance -> Firmware Upgrade and do it now if you haven’t done so already.

Firmware Upgrade

Finally there’s Maintenance -> Configuration. Here you can backup the configuration settings you just made and download an archive to your computer. Doesn’t hurt to do that.

Configuration

So much for the modem. There’s more things it can do but they are mostly only relevant in router mode (and sometimes even then only when you have special requirements).

OPNsense dialup

With the modem fully configured and working, it’s time to configure OPNsense to do the DSL dialup. I chose to rename the first network interface from WAN to PPPoE, but that’s only a name. You need to go to Interfaces -> PPPoE (or whatever yours is called) and change the IPv4 Configuration type from DHCP to PPPoE (unless you have an IPv6-only line of course in which case you’d configure that instead).

OPNsense WAN connection configuration

Further down enter the username and password for the PPPoE connection. Check the documents you got from your ISP, they should be on there somewhere. If they aren’t, ask for them.

OPNsense PPPoE configuration

And that’s all. Save your changes and if everything is correct, OPNsense will do the dialup and establish an Internet connection! Much better now that a trusted device does this, isn’t it?

Conclusion

My new setup is not perfect. Ideally I’d make my OPNsense machine deal with the IP telephony, too. Before even attempting that I will however need to do a lot of reading upfront. So there’s another long-term goal.

Nevertheless this was a change for the better. I made another step in reclaiming my own network. So far I’ve been running this setup for a month and did not face any problems. There has been a short power outage once: After power was back, the APU and the modem booted and before long OPNsense had re-established the connection and I was online again.

What’s next?

The next article will be about building custom packages on OPNsense (since it’s a somewhat involved topic it will probably be split into two posts, though).

FreeBSD router take 2 (pt. 3): Excursion – De-hardening OPNsense for 2022?

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/bsd_router_take_2_pt3.gmi

After OPNsense announced that they would rebase on vanilla FreeBSD instead of going on with HardenedBSD, I wrote the previous article on what “security” means when it comes to networked devices that are connected to the Internet. It also took a look at the fields where FreeBSD is doing pretty good. There’s also the other side of the coin however. Being a person who really likes FreeBSD and enjoys working with it, this article is not one that I looked forward to writing. But FreeBSD is not all sunshine and roses (who would have thought?). And people should be aware of that to make an educated choice. So here we go.

Defaults / (lack of) exploit mitigations

It’s the last two points from the previous article’s list that FreeBSD admittedly sucks at. As discussed there, you can make your FreeBSD systems a lot more secure than they are after a default installation. But FreeBSD does not believe in a “one size fits all” security concept. Truth be told, I’d also be very skeptical about such a concept. Either it will be so general that it’s not worth even laughing about or it will be highly theoretical and probably collide with real deployments pretty soon.

FreeBSD provides a lot of documentation to help you understand the system. However only you know your specific use case and therefore only you can put together the best possible security concept. I strongly support this way of thinking. It is true however that FreeBSD could do a lot better with regards to basic security. Why isn’t a firewall active by default? If it was that would probably be ipfw with a simplistic rule set. One of the first things that I’d do on a new installation would be disabling it and configuring pf instead. And if somebody really has a use case where there should be no firewall – well, nobody would stop you from just disabling it. Coming with any firewall enabled by default would neither limit your choices nor would it make FreeBSD unfit for any sensible scenario. Making a change like that has POLA implications, though (FreeBSD’s “Policy of least astonishment”). But if the will was there, a way would be found.

There’s other things that make it really hard to come up with an excuse for. Frankly speaking: When it comes to mitigation techniques, FreeBSD is hardly a modern operating system. Linux, Windows – basically every common system you can name did a whole lot of work in that area over the last two decades. In comparison FreeBSD did almost nothing. Unfortunately there is not much more to say about that.

And one of the few things that were done, wasn’t done right according to some security researchers. FreeBSD claims to support ASLR (Address space layout randomization). It’s not enabled by default, but it’s there:

# sysctl -a | grep aslr
kern.elf32.aslr.stack_gap: 3
kern.elf32.aslr.honor_sbrk: 1
kern.elf32.aslr.pie_enable: 0
kern.elf32.aslr.enable: 0
kern.elf64.aslr.stack_gap: 3
kern.elf64.aslr.honor_sbrk: 1
kern.elf64.aslr.pie_enable: 0
kern.elf64.aslr.enable: 0
vm.aslr_restarts: 0

The idea of this mitigation technique is to randomly arrange data of processes in memory to make it harder for an attacker to hit a targeted function. In general ASLR’s effectiveness as a mitigation has been doubted by a lot of people who are into security. Then again, it does not come with a high cost and so it’s often seen as a baseline of protection. It’s been standard in OpenBSD since 2003, in Linux since 2005 and in OS X as well as in Windows since 2007. NetBSD was a bit late to the party and only implemented it in 2009. Heck, even Oracle Solaris adopted ASLR in 2012. And FreeBSD? Totally took their time. Got it first in 12.1 which was released in late 2019. But better late than never and maybe the wait was worth it if we got a superior implementation for that?

There’s one problem, though: FreeBSD’s implementation has been criticized as half-baked… According to Shawn Webb it’s not even ASLR but ASR. So we got what we got extremely late, it’s disabled by default and even if you enable it it’s a pretty weak form of what is considered a very basic mitigation. This does not make FreeBSD look too good.

Is FreeBSD well-maintained?

Let’s poke another pain-point, shall we? FreeBSD is much, much less active in cleaning up their system than OpenBSD for example. If you take a look at the source repository, you won’t have to search too long to find things that are not all that pretty. Here’s one example of a commit updating the comcontrol manpage.

The commit removes a reference to the sio(4) interface in the comcontrol(8) manpage. This change is available in the recently released FreeBSD 13.0, older supported releases 12.2 and 11.4 still mention the interface. The thing is that sio(4) was removed from the GENERIC kernel in 2008 – which means that the manpage change that finally shipped this year (and for the very latest version only!) could easily have been shipped with FreeBSD 8.0 over ten years ago…

Want one more? Have a look here. Meet “pnpinfo” which according to the manpage “reports information about Plug-n-Play ISA devices“! Hasn’t been touched in over ten years and is very obviously completely obsolete. It’s not built by default anymore, can’t be built manually either (due to a missing system header file) – but it’s still there in the source tree. It looks like it was still part of FreeBSD 10.0 (early 2014) but removed for 10.1 (late 2014). Till the end, the pnpinfo(8) manpage referenced pnp(4) which in turn had already been removed in FreeBSD 4.6 (2002)!

Right, this is nitpicking around very minor issues. Basically every project has dusty corners and when it is the size of FreeBSD, it would be close to a miracle to not have any. Still it’s only two easy to find examples out of many that show one thing: There’s room for improvement. Plenty actually. But while *BSD prides itself in good documentation, little leftovers like this don’t have such a huge impact on security. Could really be much worse right? For example if little to no maintenance was being done on very important system components responsible for, say secure authentication?

I can hear anything from deep sighs to screams of agony from readers familiar with FreeBSD even before I put a link to Heimdal in the base system here…

Really – FreeBSD ships with Heimdal 1.5.2 in the base system… This version was released in 2012 (!!) and nobody should have trouble believing me that there’s a bunch of nasty CVEs for it. Right, everybody knows that you should never use kerberos from the base system. If you need it, always install either security/heimdal or security/krb5-$version from ports or packages. That way you’ll get versions that are up to date. But honestly: Why the heck is that ancient base system version even there? Nobody should have used it in almost a decade! What’s the point in having a trap like that lurking in base? To see if unsuspecting users might fall into it and make an acquaintance of the poisoned spears at the bottom? That’ll teach them a valuable lesson, eh? No, sorry. No point in even trying to whitewash this. It is just hideous and a real disgrace.

And then there’s of course the recent turmoil around the flawed Wireguard implementation that almost made it into FreeBSD 13. If you don’t know what I’m talking about, consider skimming over this Ars Technica article.

It is actually not a good article and I expected more of Jim Salter; he does a podcast called “2.5 admins” together with FreeBSD developer Allan Jude and they discussed the topic a couple of days before Salter wrote the linked article, forgetting some of the important things and concentrating on minor matters to have “a good story”… It will introduce you to the drama, however. Keep in mind that Jim pretends that the flawed code was “probably” only removed because the original Wireguard inventor intervened even though FreeBSD developers were looking at the code and there already were people unhappy with it (whereas he denies that the even more recent happenings around Linux and the University of Minnesota showed that things in Linux world are also far from perfect).

Bottom line: There’s all kinds of problems in FreeBSD. From small cosmetics to heavy-duty stuff. But FreeBSD is an Open Source project. If you think about contributing fixes (even for the very simple things): By all means do so! It’s not that FreeBSD wants to be in the sorry state it is in regarding certain areas. The project is taking new contributions with open arms. You’d help make the world a little bit better for many people. And there’s plenty of valuable skills to acquire if you choose to go down that road. Doc committers in FreeBSD are equal in their rights to ports and source committers, by the way. If you’ve got a bit of time for it and an interest in tech (you’re reading articles like this not because you don’t care at all, do you?) seriously consider it.

HardenedBSD

At this point the sunshine that the previous article may have shown is probably gone and there are some pretty dark clouds in the sky. Don’t let the problems that I pointed out here scare you away. Remember that the above was written by a FreeBSD user – not a former user. Everything added up, FreeBSD is a decent platform that’s not worse than any other. In fact it has a lot of advantages that help accepting some of the disadvantages. Be aware of the ugly part, though. It might bite you otherwise.

But is this a god-given situation that we cannot do anything about? Is it either the really nice features and sane structure of FreeBSD or better mitigations but much less overall usefulness of OpenBSD (alternatively the better mitigations but the chaotic mess that is Linux these days)? Fortunately not: Enter HardenedBSD.

Have a look at this image to get an idea of what HardenedBSD is doing:

HardenedBSD feature comparison

It’s only four security features listed there that OpenBSD has but HardenedBSD doesn’t. Of course the comparison is not complete, missing out a several good things in OpenBSD like e.g. pledge. However HardenedBSD also has a lot that go even further than what OpenBSD does.

And that’s really, really impressive. Keep in mind that HardenedBSD is basically FreeBSD with a ton of security improvements to it: It has ZFS, jails and all the good stuff. It’s a bit less convenient to use (e.g. you will have to understand additional tools like secadm to toggle certain mitigation features on or off for specific programs). It offers you the means to make system administration a fair bit more cumbersome – while making life terribly hard for attackers. If you are serious about security and accept that there is no free lunch, you’re willing to endure the additional restrictions for a huge gain in hardening your system.

HardenedBSD is a hardened but not a hard fork of FreeBSD. It tracks upstream FreeBSD and merges new code from there. The project also aims to develop security features outside of FreeBSD but to ultimately give the changes back. This would be a huge gain for security-focused FreeBSD users. A very small project however has also very little chance of getting FreeBSD to accept proposed hardening techniques. For that reason HardenedBSD needs every bit of support it can get.

For some time, HardenedBSD also had LibreSSL in base instead of OpenSSL. They had to switch back for the simple reason that the team was to small to keep up with the work required for such an invasive change along with all the other security improvements. And now that OPNsense has announced to ditch HardenedBSD, it will lose some more badly needed support.

So is it a hopeless case? Well, not quite. OPNsense was definitely the most prominent user of HardenedBSD but certainly not the only one. There are people and companies using it. There is being research done with it (see the e.g. this bunker jails article).

Co-founder Shawn Webb also managed to get a foundation started for it and even to attract an impressive amount of donations last year. I’d say that $13,000 instead of 11,000 they had aimed for is not bad at all! Especially if you compare it to the NetBSD foundation which only managed to get about 24,000 of their 50,000 goal even though they are a much older and bigger project.

I’ve been thinking about using HardenedBSD instead of FreeBSD when I build my next workstation. I’ll probably also use it when I reinstall my server and see how that goes. Both will probably things to write about here on the Neunix Gemlog.

Does leaving HardenedBSD make (OPN)sense?

Decisions like this are always a tradeoff and I’m not under the impression that the OPNsense team made this one without carefully considering the matter. In short-term I think that tracking mainstream FreeBSD will definitely benefit OPNsense. Here’s a couple of reasons:

  • It makes development easier in general
  • It will speed up adoption of newer releases
  • It will free resources (e.g. currently the team has to backport fixes to a no longer supported FreeBSD release)
  • It makes debugging easier
  • It might attract additional contributors familiar with FreeBSD but not HardenedBSD

Sounds good, right? If you’re willing to sacrifice the additional hardening of HardenedBSD it sure does. And I think that most people would in fact prefer to go down that route.

IMHO OPNsense is hurting itself in the long run, though. The major reason for ditching HardenedBSD is that it is too much of a niche platform after all. With OPNsense leaving it, it will become even more niche. It is a very important project to eventually take FreeBSD into the right direction. Let’s not underestimate the gem that we have here! Trying to increase adoption would be what we should be doing, not decreasing it further.

But I don’t want to challenge the decision that has been made, write a petition and bring unrest to the community. What OPNsense needs is to continue evolving for the better. One goal that aligns perfectly with the new strategy is getting rid of some more quirks that OPNsense inherited from pfSense and rather doing things like FreeBSD does. This would benefit everybody.

And who knows: Perhaps we’ll see something like “HardenedSense” in the future? Not as a fork but as a community build for people who prefer to stick with a hardened system for their packet filter needs. I hope that this is food for thought for some readers. Maybe we can start a discussion over at the forums or so. If there’s anybody interested in this, please let me know.

Why not OpenBSD?

Following the announcement of OPNsense to part ways with HardenedBSD, some users over on Reddit proposed to rebase on OpenBSD instead. Let’s consider this for a moment.

OpenBSD is generally regarded as a very, very secure operating system. It has a great lot of mitigations in place, a nice and clean codebase and a reasonable-sized community. That’s certainly appealing. People also frequently mention that it has a much newer version of Pf which would be very much beneficial for a project like OPNsense.

There’s a couple of reasons why this is not as good an idea as it seems, though. I actually like what the OpenBSD people are doing. No, truth be told, I admire their security first stance and the fact that they are willing to take it to the extreme anytime. But… Exactly this makes it the wrong choice for anything like OPNsense:

  • Performance is not a primary goal for OpenBSD. If you want a top-notch router, you’ve ruled it out.
  • OpenBSD is a research OS. You can use it in production but you have to live with things like NO ABI stability whatsoever.
  • There’s a lot less software packaged for OpenBSD.
  • OpenBSD does not provide safe data storage, they don’t have any next-gen filesystem.

Let’s also address the misconception of “newer Pf on OpenBSD”: This is not true. Pf originated in OpenBSD when they dropped (due to licensing issues) IPF which they used before and replaced it with their own packet filter. Pf was later ported to FreeBSD (and NetBSD). After those ports happened, OpenBSD continued to improve Pf. One thing that they did was revising the syntax. FreeBSD did not sync their Pf with OpenBSD anymore – but for a good reason! They had improved their version of Pf to make it perform much better with multi-core CPUs. Contributing those changes back to OpenBSD was hopeless since OpenBSD was largely not SMP-capable at that time. For that reason Pf on OpenBSD and Pf on FreeBSD diverged, up to the point where merging newer changes from OpenBSD was simply not feasible anymore.

So it’s not that OpenBSD has “newer PF” – it’s more like both OpenBSD and FreeBSD have distinct versions of Pf that are actively developed but are quite different despite the common name. Rebasing OPNsense on OpenBSD would not give the users a much better Pf. In fact the major user-visible advantage of OpenBSD’s version of Pf – i.e. the simpler syntax – would not even be user-visible on OPNsense as people use the GUI to create their rules! It would on the contrary mean that code changes would be required so that the OPNsense application responsible for the rules would generate the rules in the new syntax expected by OpenBSD’s Pf.

There would also be a lot of other things to change. OpenBSD’s networking works quite a bit differently (e.g. the system’s hostname goes into /etc/myname instead of into /etc/rc.conf as used in FreeBSD). The init system is slightly different. Packaging works very differently (not using /usr/local for example and the package managers are simply worlds apart). And so on.

Conclusion

FreeBSD is a solid operating system that’s doing well overall but is severely lacking in certain areas. HardenedBSD offers all the benefits of FreeBSD without a lot of the weaknesses and is an innovating force when it comes to strong security. OPNsense leaving HardenedBSD behind is a sensible choice considering OPNsense alone but a very unfortunate move for the FreeBSD ecosystem as a whole. OpenBSD is not the right base for OPNsense either.

If you care for FreeBSD and security, please support HardenedBSD. Let’s keep it going strong – maybe there’s the chance of having a community edition of 22.1 and onward that’s still going to be based on HardenedBSD if there is enough interest.

FreeBSD router take 2 (pt. 2): Excursion – FreeBSD and security

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/bsd_router_take_2_pt2.gmi

After I completed the previous article, Franco Fichtner announced that OPNsense and HardenedBSD will be parting ways.

I’m happy to see that they are parting ways in good terms. So there at least were no ugly things going on behind the curtain. The explanations given in the announcement are interesting and I’d say that they make sense. This is a pretty massive change, though. And since (thanks to the friendly Web UI) OPNsense is used by a lot of people who do not have a FreeBSD background, I’d like to explain in a bit more detail what the actual situation is like.

In the first series of posts, the second one was an excursion on using the serial console. This time we’re going to take a look at the broad topic of security.

What is “security”?

FreeBSD has had incredibly talented security officers like Colin Percival, founder of the Tarsnap backup company. His company’s motto is Online backups for the truly paranoid – and it lives up to that. Thanks to him and many great people in the security team, FreeBSD has built up a fairly good reputation regarding security with a lot of people.

There are other voices, too. For example one former FreeBSD user who switched to OpenBSD is industriously working on making FreeBSD look bad. Here’s the homepage where he keeps track of all the things he thinks FreeBSD does wrong.

So which claim is true then? Is FreeBSD doing pretty well or is it downright horrible?

Both of them. And neither. Oh well… It’s a bit too complicated to give a plain and simple answer. So let’s think about what “security” actually means for a moment before returning to judge FreeBSD’s performance in that area.

There are multiple aspects to security. To take the whole situation into consideration means to admit that we’re in one giant mess right now!

Living in a nightmare

We absolutely depend on today’s technology. Think about replacing “the Internet” for example. Even if you have this exceptionally great idea and can provide a concept that is totally sound – how do you think we could get there? Millions of enterprises require the Internet as we know it to continue working as it does. The chances of succeeding with establishing something better that would run in parallel? Basically nonexistent. Nobody could pay for such a huge project! And even if it were to magically appear and be available tomorrow (somehow production-ready by day one), how do you think getting a critical mass of businesses to adapt it?

Incrementally improving what we have is hard enough. If you disagree just think about disabling anything but TLS 1.3 on your employer’s webservers. You, dear reader, probably are ready for such a change, I wouldn’t doubt that. But are all your customers? And that’s only one example of… many.

While being condemned to never being able to “re-invent the wheel” in large scale is unfortunate, it’s not catastrophic. What is catastrophic however is that the very foundations of the technology we’re using today were very much over-credulous from today’s point of view. It’s perfectly reasonable not designing network protocols for security when you don’t think of potential offenders because your network is either limited to one institution or basically to a couple of universities! We’ve outgrown those innocent times for a long time now however. The Internet is a war zone.

If you feel brave, join us and participate in our Gemini experiment (see top of this article). Get a Gemini client, read this article over that protocol. Ideally get your own gemlog started and share original content with the world. While we’re not even dreaming of replacing the Internet with something better – even the very act of challenging the Web alone by providing an alternative for like-minded people who loathe superfluous complexity, is an Herculean effort in and of itself.

But back on topic. Retrofitting security into existing technology that’s already in production use is incredibly hard. Especially if you are supposed to NOT break the former! And if it wasn’t hard enough, this scenario also comes with the curse of optional security which is another pretty sad story by itself… Your DNS server probably supports DNSSEC by now. But does it use DANE (mine doesn’t, yet…)? And how many of the more popular nameservers on the Internet do? I mean, it’s been almost a decade since it was introduced. We need regular “DNS flag days” to force people adapting somewhat acceptable DNS standards. How much can we expect optional security features to come to wide-spread use?

And if all of that wasn’t bad enough, in 2018 we learned that the one most important CPU architecture (x86) has a flawed design that dates two decades back… If you want to refresh your knowledge of what Meltdown and Spectre are, here’s an excellent read (explained so that each and every layman can understand it) by the aforementioned Colin Percival: Some thoughts on spectre and meltdown.

Another point is that while almost everybody tends to agree that security is “important”, few want to actually spend money or effort to improve it. Reading last year’s FOSS Contributor Survey of the Linux Foundation gives you an idea of how bad the state of affairs really is.

It’s 2.3 percent (on average!) of a developer’s time that is spent on work to fix security-related issues in their projects. If you assume a paid developer working 9-to-5, that’s about 11 minutes per workday, adding up to not even an hour a week… It’s a clear trend to rather work on exciting new features than fixing bugs in existing code. Working on security-related bugs is even less popular. How many people with such a mindset would you expect to proactively audit their code for flaws regarding security?

Security as “surviving in a deadly environment”

We set sail and started exploring silent waters around the coast with a pretty much experimental boat. It has been a very exciting ride – because at some point we realized that we were no longer near the coast but somewhere in the middle of the ocean. So what was really a nice toy before has turned into a necessity for us to survive. Quite a while ago we noticed that we were in stormy water and that we had to constantly fix our humble boat when the force of another tide tried to smash it! We’re constantly fixing new leaks and fighting off dangers that nobody really anticipated. And to make it even worse, while a lot of people were interested in tuning our boat for performance, they didn’t want to see that the water around us started boiling. Today we’re surrounded by lava. There is no lifeboat. If you shipwreck, you’re dead.

That picture should have either reminded you of what you already knew – or have been eye-opening regarding our situation. There is no need to panic (that wouldn’t be helpful), but don’t fall into the trap of simply dismissing the shadowy dangers. They are very much real! So whatever helps your employer survive in this situation could be thought of as a security feature.

There is no single panacea but combining a lot of security features can drastically lower the threats. Here are some important bits:

  • Respond to discovered vulnerabilities in a timely manner
  • Offer the latest versions of software or backport fixes
  • Provide means that harden the system
  • Develop mitigations for when (not if!) your first line of defense is breached
  • Make it as easy as possible for the user to secure the system

Where FreeBSD does mostly well

When it comes to patching base system vulnerabilities, FreeBSD is generally doing well (there’s no point in denying that things do not always work like they should so that blakkheim can point a finger at it). If you’ve never read a security advisory as published by the FreeBSD project, I suggest you do so at least once to get an idea. Like the latest one.

As you can see, the security team does not only silently fix bugs but even goes an extra mile, doing a write-up for the interested reader. For years I’ve been very happy with those; I’m not a developer with sharp C skills and deep knowledge of exactly how programs work, but I can always understand what’s going on there. I’m not spiteful enough to ask you to compare them with OpenBSD’s errata which are… very bare-bones.

If you take a closer look at the example provided here, you can see that among the versions that received a fix is 13.0-RC5-p1. That’s right: This means that they even cared enough to fix it for a Release Candidate that has a lifecycle of two weeks! And not only that, they even provided binary updates for that even though people on RC5 could just be expected to update to 13.0-RELEASE only a couple of days later. I’d say that this is nothing short of very commendable acting.

Regarding packages that are not vulnerable, the situation with FreeBSD is a mixed bag. There are quite a few unmaintained ports stuck at older, insecure versions. Common software is usually pretty recent. To give you an idea (and so that you don’t simply have to take my word for it), let’s compare FreeBSD and Ubuntu 21.04. FreeBSD currently has a port count of slightly above 31,000 whereas Ubuntu offers just short of 33,000 packages. A little over 6,000 are outdated on FreeBSD, for Ubuntu it’s over 8,500. For ports beginning with the letter “A”, FreeBSD has 9 ports with versions that contain a known vulnerability (with a CVE) that could be fixed by updating to a newer version whereas Ubuntu has only 3. Same thing for ports that begin with “Z”: 1 vulnerable port that could be fixed by updating to a newer version on FreeBSD, 2 such packages on Ubuntu.

Just so nobody claims I’d selectively present data to support either story with it, here’s a table for package CVEs of all starting letters:

Starting letter FreeBSD # Ubuntu 21.04 #
A 9 3
B 3 3
C 4 7
D 2 4
E 3 0
F 4 1
G 9 5
H 2 2
I 3 4
J 8 3
K 0 1
L 12 13
M 8 3
N 2 18
O 4 6
P 18 17
Q 0 0
R 8 13
S 8 10
T 5 7
U 2 2
V 2 2
W 3 2
X 3 5
Y 1 0
Z 1 2
TOTAL 125 133

I think it’s safe to say: FreeBSD does pretty good in this field, too! Especially if you take into consideration that most of FreeBSD’s ports are done entirely by volunteers and that while software usually “just works” on Linux there’s often some more work required to make it work on other operating systems! (Of course I’m aware that I’m just scratching the surface here and a deeper analysis would be nice – but that would definitely take its own article.)

Let’s talk about means of hardening the system. There’s a lot you can do to harden a system that was installed using the default options. For a while now (I think starting with 11.0) FreeBSD offers a hardening dialog in the installer, allowing for really simple improvement of the defaults. This is one thing that blakkheim prefers to ignore: Yes, /tmp is not cleared by default, but FreeBSD can do that if you want it to and it’s not hard to make it do that.

Yes, hardening a FreeBSD system is not something you can expect the junior admin to master in a couple of hours. But that doesn’t mean that it’s impossible to do. With securelevels and file flags, FreeBSD gives you a powerful tool for increased security. Capsicum and casper are two more things you can take a look at and start making use of. Taking advantage of jailing applications is another great way to make your infrastructure more secure by confining possible intruders and further limiting the damage they can do. FreeBSD expects you to do all that and more, depending on what your security requirements are.

Definitely have a look at security(7). To quote from it:

A little paranoia never hurts. As a rule, a sysadmin can add any number of security features as long as they do not affect convenience, and can add security features that do affect convenience with some added thought. Even more importantly, a security administrator should mix it up a bit — if you use recommendations such as those given by this manual page verbatim, you give away your methodologies to the prospective attacker who also has access to this manual page.

Does that really sound like it’s written by people who do not care for security at all as our friend blakkheim wants you to believe?

So far for the good part. The ugly side of FreeBSD will be covered next time as this article is already way too long. Thanks for reading!

What’s next?

The next post will briefly discuss FreeBSD’s security weaknesses and how HardenedBSD fits into the picture. I’ll also address the “rebase it on OpenBSD!” suggestion some people have made.