FreeBSD router take 2 (pt. 3): Excursion – De-hardening OPNsense for 2022?

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/bsd_router_take_2_pt3.gmi

After OPNsense announced that they would rebase on vanilla FreeBSD instead of going on with HardenedBSD, I wrote the previous article on what “security” means when it comes to networked devices that are connected to the Internet. It also took a look at the fields where FreeBSD is doing pretty good. There’s also the other side of the coin however. Being a person who really likes FreeBSD and enjoys working with it, this article is not one that I looked forward to writing. But FreeBSD is not all sunshine and roses (who would have thought?). And people should be aware of that to make an educated choice. So here we go.

Defaults / (lack of) exploit mitigations

It’s the last two points from the previous article’s list that FreeBSD admittedly sucks at. As discussed there, you can make your FreeBSD systems a lot more secure than they are after a default installation. But FreeBSD does not believe in a “one size fits all” security concept. Truth be told, I’d also be very skeptical about such a concept. Either it will be so general that it’s not worth even laughing about or it will be highly theoretical and probably collide with real deployments pretty soon.

FreeBSD provides a lot of documentation to help you understand the system. However only you know your specific use case and therefore only you can put together the best possible security concept. I strongly support this way of thinking. It is true however that FreeBSD could do a lot better with regards to basic security. Why isn’t a firewall active by default? If it was that would probably be ipfw with a simplistic rule set. One of the first things that I’d do on a new installation would be disabling it and configuring pf instead. And if somebody really has a use case where there should be no firewall – well, nobody would stop you from just disabling it. Coming with any firewall enabled by default would neither limit your choices nor would it make FreeBSD unfit for any sensible scenario. Making a change like that has POLA implications, though (FreeBSD’s “Policy of least astonishment”). But if the will was there, a way would be found.

There’s other things that make it really hard to come up with an excuse for. Frankly speaking: When it comes to mitigation techniques, FreeBSD is hardly a modern operating system. Linux, Windows – basically every common system you can name did a whole lot of work in that area over the last two decades. In comparison FreeBSD did almost nothing. Unfortunately there is not much more to say about that.

And one of the few things that were done, wasn’t done right according to some security researchers. FreeBSD claims to support ASLR (Address space layout randomization). It’s not enabled by default, but it’s there:

# sysctl -a | grep aslr
kern.elf32.aslr.stack_gap: 3
kern.elf32.aslr.honor_sbrk: 1
kern.elf32.aslr.pie_enable: 0
kern.elf32.aslr.enable: 0
kern.elf64.aslr.stack_gap: 3
kern.elf64.aslr.honor_sbrk: 1
kern.elf64.aslr.pie_enable: 0
kern.elf64.aslr.enable: 0
vm.aslr_restarts: 0

The idea of this mitigation technique is to randomly arrange data of processes in memory to make it harder for an attacker to hit a targeted function. In general ASLR’s effectiveness as a mitigation has been doubted by a lot of people who are into security. Then again, it does not come with a high cost and so it’s often seen as a baseline of protection. It’s been standard in OpenBSD since 2003, in Linux since 2005 and in OS X as well as in Windows since 2007. NetBSD was a bit late to the party and only implemented it in 2009. Heck, even Oracle Solaris adopted ASLR in 2012. And FreeBSD? Totally took their time. Got it first in 12.1 which was released in late 2019. But better late than never and maybe the wait was worth it if we got a superior implementation for that?

There’s one problem, though: FreeBSD’s implementation has been criticized as half-baked… According to Shawn Webb it’s not even ASLR but ASR. So we got what we got extremely late, it’s disabled by default and even if you enable it it’s a pretty weak form of what is considered a very basic mitigation. This does not make FreeBSD look too good.

Is FreeBSD well-maintained?

Let’s poke another pain-point, shall we? FreeBSD is much, much less active in cleaning up their system than OpenBSD for example. If you take a look at the source repository, you won’t have to search too long to find things that are not all that pretty. Here’s one example of a commit updating the comcontrol manpage.

The commit removes a reference to the sio(4) interface in the comcontrol(8) manpage. This change is available in the recently released FreeBSD 13.0, older supported releases 12.2 and 11.4 still mention the interface. The thing is that sio(4) was removed from the GENERIC kernel in 2008 – which means that the manpage change that finally shipped this year (and for the very latest version only!) could easily have been shipped with FreeBSD 8.0 over ten years ago…

Want one more? Have a look here. Meet “pnpinfo” which according to the manpage “reports information about Plug-n-Play ISA devices“! Hasn’t been touched in over ten years and is very obviously completely obsolete. It’s not built by default anymore, can’t be built manually either (due to a missing system header file) – but it’s still there in the source tree. It looks like it was still part of FreeBSD 10.0 (early 2014) but removed for 10.1 (late 2014). Till the end, the pnpinfo(8) manpage referenced pnp(4) which in turn had already been removed in FreeBSD 4.6 (2002)!

Right, this is nitpicking around very minor issues. Basically every project has dusty corners and when it is the size of FreeBSD, it would be close to a miracle to not have any. Still it’s only two easy to find examples out of many that show one thing: There’s room for improvement. Plenty actually. But while *BSD prides itself in good documentation, little leftovers like this don’t have such a huge impact on security. Could really be much worse right? For example if little to no maintenance was being done on very important system components responsible for, say secure authentication?

I can hear anything from deep sighs to screams of agony from readers familiar with FreeBSD even before I put a link to Heimdal in the base system here…

Really – FreeBSD ships with Heimdal 1.5.2 in the base system… This version was released in 2012 (!!) and nobody should have trouble believing me that there’s a bunch of nasty CVEs for it. Right, everybody knows that you should never use kerberos from the base system. If you need it, always install either security/heimdal or security/krb5-$version from ports or packages. That way you’ll get versions that are up to date. But honestly: Why the heck is that ancient base system version even there? Nobody should have used it in almost a decade! What’s the point in having a trap like that lurking in base? To see if unsuspecting users might fall into it and make an acquaintance of the poisoned spears at the bottom? That’ll teach them a valuable lesson, eh? No, sorry. No point in even trying to whitewash this. It is just hideous and a real disgrace.

And then there’s of course the recent turmoil around the flawed Wireguard implementation that almost made it into FreeBSD 13. If you don’t know what I’m talking about, consider skimming over this Ars Technica article.

It is actually not a good article and I expected more of Jim Salter; he does a podcast called “2.5 admins” together with FreeBSD developer Allan Jude and they discussed the topic a couple of days before Salter wrote the linked article, forgetting some of the important things and concentrating on minor matters to have “a good story”… It will introduce you to the drama, however. Keep in mind that Jim pretends that the flawed code was “probably” only removed because the original Wireguard inventor intervened even though FreeBSD developers were looking at the code and there already were people unhappy with it (whereas he denies that the even more recent happenings around Linux and the University of Minnesota showed that things in Linux world are also far from perfect).

Bottom line: There’s all kinds of problems in FreeBSD. From small cosmetics to heavy-duty stuff. But FreeBSD is an Open Source project. If you think about contributing fixes (even for the very simple things): By all means do so! It’s not that FreeBSD wants to be in the sorry state it is in regarding certain areas. The project is taking new contributions with open arms. You’d help make the world a little bit better for many people. And there’s plenty of valuable skills to acquire if you choose to go down that road. Doc committers in FreeBSD are equal in their rights to ports and source committers, by the way. If you’ve got a bit of time for it and an interest in tech (you’re reading articles like this not because you don’t care at all, do you?) seriously consider it.

HardenedBSD

At this point the sunshine that the previous article may have shown is probably gone and there are some pretty dark clouds in the sky. Don’t let the problems that I pointed out here scare you away. Remember that the above was written by a FreeBSD user – not a former user. Everything added up, FreeBSD is a decent platform that’s not worse than any other. In fact it has a lot of advantages that help accepting some of the disadvantages. Be aware of the ugly part, though. It might bite you otherwise.

But is this a god-given situation that we cannot do anything about? Is it either the really nice features and sane structure of FreeBSD or better mitigations but much less overall usefulness of OpenBSD (alternatively the better mitigations but the chaotic mess that is Linux these days)? Fortunately not: Enter HardenedBSD.

Have a look at this image to get an idea of what HardenedBSD is doing:

HardenedBSD feature comparison

It’s only four security features listed there that OpenBSD has but HardenedBSD doesn’t. Of course the comparison is not complete, missing out a several good things in OpenBSD like e.g. pledge. However HardenedBSD also has a lot that go even further than what OpenBSD does.

And that’s really, really impressive. Keep in mind that HardenedBSD is basically FreeBSD with a ton of security improvements to it: It has ZFS, jails and all the good stuff. It’s a bit less convenient to use (e.g. you will have to understand additional tools like secadm to toggle certain mitigation features on or off for specific programs). It offers you the means to make system administration a fair bit more cumbersome – while making life terribly hard for attackers. If you are serious about security and accept that there is no free lunch, you’re willing to endure the additional restrictions for a huge gain in hardening your system.

HardenedBSD is a hardened but not a hard fork of FreeBSD. It tracks upstream FreeBSD and merges new code from there. The project also aims to develop security features outside of FreeBSD but to ultimately give the changes back. This would be a huge gain for security-focused FreeBSD users. A very small project however has also very little chance of getting FreeBSD to accept proposed hardening techniques. For that reason HardenedBSD needs every bit of support it can get.

For some time, HardenedBSD also had LibreSSL in base instead of OpenSSL. They had to switch back for the simple reason that the team was to small to keep up with the work required for such an invasive change along with all the other security improvements. And now that OPNsense has announced to ditch HardenedBSD, it will lose some more badly needed support.

So is it a hopeless case? Well, not quite. OPNsense was definitely the most prominent user of HardenedBSD but certainly not the only one. There are people and companies using it. There is being research done with it (see the e.g. this bunker jails article).

Co-founder Shawn Webb also managed to get a foundation started for it and even to attract an impressive amount of donations last year. I’d say that $13,000 instead of 11,000 they had aimed for is not bad at all! Especially if you compare it to the NetBSD foundation which only managed to get about 24,000 of their 50,000 goal even though they are a much older and bigger project.

I’ve been thinking about using HardenedBSD instead of FreeBSD when I build my next workstation. I’ll probably also use it when I reinstall my server and see how that goes. Both will probably things to write about here on the Neunix Gemlog.

Does leaving HardenedBSD make (OPN)sense?

Decisions like this are always a tradeoff and I’m not under the impression that the OPNsense team made this one without carefully considering the matter. In short-term I think that tracking mainstream FreeBSD will definitely benefit OPNsense. Here’s a couple of reasons:

• It makes development easier in general
• It will speed up adoption of newer releases
• It will free resources (e.g. currently the team has to backport fixes to a no longer supported FreeBSD release)
• It makes debugging easier
• It might attract additional contributors familiar with FreeBSD but not HardenedBSD

Sounds good, right? If you’re willing to sacrifice the additional hardening of HardenedBSD it sure does. And I think that most people would in fact prefer to go down that route.

IMHO OPNsense is hurting itself in the long run, though. The major reason for ditching HardenedBSD is that it is too much of a niche platform after all. With OPNsense leaving it, it will become even more niche. It is a very important project to eventually take FreeBSD into the right direction. Let’s not underestimate the gem that we have here! Trying to increase adoption would be what we should be doing, not decreasing it further.

But I don’t want to challenge the decision that has been made, write a petition and bring unrest to the community. What OPNsense needs is to continue evolving for the better. One goal that aligns perfectly with the new strategy is getting rid of some more quirks that OPNsense inherited from pfSense and rather doing things like FreeBSD does. This would benefit everybody.

And who knows: Perhaps we’ll see something like “HardenedSense” in the future? Not as a fork but as a community build for people who prefer to stick with a hardened system for their packet filter needs. I hope that this is food for thought for some readers. Maybe we can start a discussion over at the forums or so. If there’s anybody interested in this, please let me know.

Why not OpenBSD?

Following the announcement of OPNsense to part ways with HardenedBSD, some users over on Reddit proposed to rebase on OpenBSD instead. Let’s consider this for a moment.

OpenBSD is generally regarded as a very, very secure operating system. It has a great lot of mitigations in place, a nice and clean codebase and a reasonable-sized community. That’s certainly appealing. People also frequently mention that it has a much newer version of Pf which would be very much beneficial for a project like OPNsense.

There’s a couple of reasons why this is not as good an idea as it seems, though. I actually like what the OpenBSD people are doing. No, truth be told, I admire their security first stance and the fact that they are willing to take it to the extreme anytime. But… Exactly this makes it the wrong choice for anything like OPNsense:

• Performance is not a primary goal for OpenBSD. If you want a top-notch router, you’ve ruled it out.
• OpenBSD is a research OS. You can use it in production but you have to live with things like NO ABI stability whatsoever.
• There’s a lot less software packaged for OpenBSD.
• OpenBSD does not provide safe data storage, they don’t have any next-gen filesystem.

Let’s also address the misconception of “newer Pf on OpenBSD”: This is not true. Pf originated in OpenBSD when they dropped (due to licensing issues) IPF which they used before and replaced it with their own packet filter. Pf was later ported to FreeBSD (and NetBSD). After those ports happened, OpenBSD continued to improve Pf. One thing that they did was revising the syntax. FreeBSD did not sync their Pf with OpenBSD anymore – but for a good reason! They had improved their version of Pf to make it perform much better with multi-core CPUs. Contributing those changes back to OpenBSD was hopeless since OpenBSD was largely not SMP-capable at that time. For that reason Pf on OpenBSD and Pf on FreeBSD diverged, up to the point where merging newer changes from OpenBSD was simply not feasible anymore.

So it’s not that OpenBSD has “newer PF” – it’s more like both OpenBSD and FreeBSD have distinct versions of Pf that are actively developed but are quite different despite the common name. Rebasing OPNsense on OpenBSD would not give the users a much better Pf. In fact the major user-visible advantage of OpenBSD’s version of Pf – i.e. the simpler syntax – would not even be user-visible on OPNsense as people use the GUI to create their rules! It would on the contrary mean that code changes would be required so that the OPNsense application responsible for the rules would generate the rules in the new syntax expected by OpenBSD’s Pf.

There would also be a lot of other things to change. OpenBSD’s networking works quite a bit differently (e.g. the system’s hostname goes into /etc/myname instead of into /etc/rc.conf as used in FreeBSD). The init system is slightly different. Packaging works very differently (not using /usr/local for example and the package managers are simply worlds apart). And so on.

Conclusion

FreeBSD is a solid operating system that’s doing well overall but is severely lacking in certain areas. HardenedBSD offers all the benefits of FreeBSD without a lot of the weaknesses and is an innovating force when it comes to strong security. OPNsense leaving HardenedBSD behind is a sensible choice considering OPNsense alone but a very unfortunate move for the FreeBSD ecosystem as a whole. OpenBSD is not the right base for OPNsense either.

If you care for FreeBSD and security, please support HardenedBSD. Let’s keep it going strong – maybe there’s the chance of having a community edition of 22.1 and onward that’s still going to be based on HardenedBSD if there is enough interest.

FreeBSD router take 2 (pt. 2): Excursion – FreeBSD and security

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/bsd_router_take_2_pt2.gmi

After I completed the previous article, Franco Fichtner announced that OPNsense and HardenedBSD will be parting ways.

I’m happy to see that they are parting ways in good terms. So there at least were no ugly things going on behind the curtain. The explanations given in the announcement are interesting and I’d say that they make sense. This is a pretty massive change, though. And since (thanks to the friendly Web UI) OPNsense is used by a lot of people who do not have a FreeBSD background, I’d like to explain in a bit more detail what the actual situation is like.

In the first series of posts, the second one was an excursion on using the serial console. This time we’re going to take a look at the broad topic of security.

What is “security”?

FreeBSD has had incredibly talented security officers like Colin Percival, founder of the Tarsnap backup company. His company’s motto is Online backups for the truly paranoid – and it lives up to that. Thanks to him and many great people in the security team, FreeBSD has built up a fairly good reputation regarding security with a lot of people.

There are other voices, too. For example one former FreeBSD user who switched to OpenBSD is industriously working on making FreeBSD look bad. Here’s the homepage where he keeps track of all the things he thinks FreeBSD does wrong.

So which claim is true then? Is FreeBSD doing pretty well or is it downright horrible?

Both of them. And neither. Oh well… It’s a bit too complicated to give a plain and simple answer. So let’s think about what “security” actually means for a moment before returning to judge FreeBSD’s performance in that area.

There are multiple aspects to security. To take the whole situation into consideration means to admit that we’re in one giant mess right now!

Living in a nightmare

We absolutely depend on today’s technology. Think about replacing “the Internet” for example. Even if you have this exceptionally great idea and can provide a concept that is totally sound – how do you think we could get there? Millions of enterprises require the Internet as we know it to continue working as it does. The chances of succeeding with establishing something better that would run in parallel? Basically nonexistent. Nobody could pay for such a huge project! And even if it were to magically appear and be available tomorrow (somehow production-ready by day one), how do you think getting a critical mass of businesses to adapt it?

Incrementally improving what we have is hard enough. If you disagree just think about disabling anything but TLS 1.3 on your employer’s webservers. You, dear reader, probably are ready for such a change, I wouldn’t doubt that. But are all your customers? And that’s only one example of… many.

While being condemned to never being able to “re-invent the wheel” in large scale is unfortunate, it’s not catastrophic. What is catastrophic however is that the very foundations of the technology we’re using today were very much over-credulous from today’s point of view. It’s perfectly reasonable not designing network protocols for security when you don’t think of potential offenders because your network is either limited to one institution or basically to a couple of universities! We’ve outgrown those innocent times for a long time now however. The Internet is a war zone.

If you feel brave, join us and participate in our Gemini experiment (see top of this article). Get a Gemini client, read this article over that protocol. Ideally get your own gemlog started and share original content with the world. While we’re not even dreaming of replacing the Internet with something better – even the very act of challenging the Web alone by providing an alternative for like-minded people who loathe superfluous complexity, is an Herculean effort in and of itself.

But back on topic. Retrofitting security into existing technology that’s already in production use is incredibly hard. Especially if you are supposed to NOT break the former! And if it wasn’t hard enough, this scenario also comes with the curse of optional security which is another pretty sad story by itself… Your DNS server probably supports DNSSEC by now. But does it use DANE (mine doesn’t, yet…)? And how many of the more popular nameservers on the Internet do? I mean, it’s been almost a decade since it was introduced. We need regular “DNS flag days” to force people adapting somewhat acceptable DNS standards. How much can we expect optional security features to come to wide-spread use?

And if all of that wasn’t bad enough, in 2018 we learned that the one most important CPU architecture (x86) has a flawed design that dates two decades back… If you want to refresh your knowledge of what Meltdown and Spectre are, here’s an excellent read (explained so that each and every layman can understand it) by the aforementioned Colin Percival: Some thoughts on spectre and meltdown.

Another point is that while almost everybody tends to agree that security is “important”, few want to actually spend money or effort to improve it. Reading last year’s FOSS Contributor Survey of the Linux Foundation gives you an idea of how bad the state of affairs really is.

It’s 2.3 percent (on average!) of a developer’s time that is spent on work to fix security-related issues in their projects. If you assume a paid developer working 9-to-5, that’s about 11 minutes per workday, adding up to not even an hour a week… It’s a clear trend to rather work on exciting new features than fixing bugs in existing code. Working on security-related bugs is even less popular. How many people with such a mindset would you expect to proactively audit their code for flaws regarding security?

Security as “surviving in a deadly environment”

We set sail and started exploring silent waters around the coast with a pretty much experimental boat. It has been a very exciting ride – because at some point we realized that we were no longer near the coast but somewhere in the middle of the ocean. So what was really a nice toy before has turned into a necessity for us to survive. Quite a while ago we noticed that we were in stormy water and that we had to constantly fix our humble boat when the force of another tide tried to smash it! We’re constantly fixing new leaks and fighting off dangers that nobody really anticipated. And to make it even worse, while a lot of people were interested in tuning our boat for performance, they didn’t want to see that the water around us started boiling. Today we’re surrounded by lava. There is no lifeboat. If you shipwreck, you’re dead.

That picture should have either reminded you of what you already knew – or have been eye-opening regarding our situation. There is no need to panic (that wouldn’t be helpful), but don’t fall into the trap of simply dismissing the shadowy dangers. They are very much real! So whatever helps your employer survive in this situation could be thought of as a security feature.

There is no single panacea but combining a lot of security features can drastically lower the threats. Here are some important bits:

• Respond to discovered vulnerabilities in a timely manner
• Offer the latest versions of software or backport fixes
• Provide means that harden the system
• Develop mitigations for when (not if!) your first line of defense is breached
• Make it as easy as possible for the user to secure the system

Where FreeBSD does mostly well

When it comes to patching base system vulnerabilities, FreeBSD is generally doing well (there’s no point in denying that things do not always work like they should so that blakkheim can point a finger at it). If you’ve never read a security advisory as published by the FreeBSD project, I suggest you do so at least once to get an idea. Like the latest one.

As you can see, the security team does not only silently fix bugs but even goes an extra mile, doing a write-up for the interested reader. For years I’ve been very happy with those; I’m not a developer with sharp C skills and deep knowledge of exactly how programs work, but I can always understand what’s going on there. I’m not spiteful enough to ask you to compare them with OpenBSD’s errata which are… very bare-bones.

If you take a closer look at the example provided here, you can see that among the versions that received a fix is 13.0-RC5-p1. That’s right: This means that they even cared enough to fix it for a Release Candidate that has a lifecycle of two weeks! And not only that, they even provided binary updates for that even though people on RC5 could just be expected to update to 13.0-RELEASE only a couple of days later. I’d say that this is nothing short of very commendable acting.

Regarding packages that are not vulnerable, the situation with FreeBSD is a mixed bag. There are quite a few unmaintained ports stuck at older, insecure versions. Common software is usually pretty recent. To give you an idea (and so that you don’t simply have to take my word for it), let’s compare FreeBSD and Ubuntu 21.04. FreeBSD currently has a port count of slightly above 31,000 whereas Ubuntu offers just short of 33,000 packages. A little over 6,000 are outdated on FreeBSD, for Ubuntu it’s over 8,500. For ports beginning with the letter “A”, FreeBSD has 9 ports with versions that contain a known vulnerability (with a CVE) that could be fixed by updating to a newer version whereas Ubuntu has only 3. Same thing for ports that begin with “Z”: 1 vulnerable port that could be fixed by updating to a newer version on FreeBSD, 2 such packages on Ubuntu.

Just so nobody claims I’d selectively present data to support either story with it, here’s a table for package CVEs of all starting letters:

Starting letter	FreeBSD #	Ubuntu 21.04 #
A	9	3
B	3	3
C	4	7
D	2	4
E	3	0
F	4	1
G	9	5
H	2	2
I	3	4
J	8	3
K	0	1
L	12	13
M	8	3
N	2	18
O	4	6
P	18	17
Q	0	0
R	8	13
S	8	10
T	5	7
U	2	2
V	2	2
W	3	2
X	3	5
Y	1	0
Z	1	2
TOTAL	125	133

I think it’s safe to say: FreeBSD does pretty good in this field, too! Especially if you take into consideration that most of FreeBSD’s ports are done entirely by volunteers and that while software usually “just works” on Linux there’s often some more work required to make it work on other operating systems! (Of course I’m aware that I’m just scratching the surface here and a deeper analysis would be nice – but that would definitely take its own article.)

Let’s talk about means of hardening the system. There’s a lot you can do to harden a system that was installed using the default options. For a while now (I think starting with 11.0) FreeBSD offers a hardening dialog in the installer, allowing for really simple improvement of the defaults. This is one thing that blakkheim prefers to ignore: Yes, /tmp is not cleared by default, but FreeBSD can do that if you want it to and it’s not hard to make it do that.

Yes, hardening a FreeBSD system is not something you can expect the junior admin to master in a couple of hours. But that doesn’t mean that it’s impossible to do. With securelevels and file flags, FreeBSD gives you a powerful tool for increased security. Capsicum and casper are two more things you can take a look at and start making use of. Taking advantage of jailing applications is another great way to make your infrastructure more secure by confining possible intruders and further limiting the damage they can do. FreeBSD expects you to do all that and more, depending on what your security requirements are.

Definitely have a look at security(7). To quote from it:

A little paranoia never hurts. As a rule, a sysadmin can add any number of security features as long as they do not affect convenience, and can add security features that do affect convenience with some added thought. Even more importantly, a security administrator should mix it up a bit — if you use recommendations such as those given by this manual page verbatim, you give away your methodologies to the prospective attacker who also has access to this manual page.

Does that really sound like it’s written by people who do not care for security at all as our friend blakkheim wants you to believe?

So far for the good part. The ugly side of FreeBSD will be covered next time as this article is already way too long. Thanks for reading!

What’s next?

The next post will briefly discuss FreeBSD’s security weaknesses and how HardenedBSD fits into the picture. I’ll also address the “rebase it on OpenBSD!” suggestion some people have made.

FreeBSD router take 2 (pt. 1): OPNsense ZFS-based installation (by converting FreeBSD)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/bsd_router_take_2_pt1.gmi

In 2017 I wrote my longest (by far) series of posts on a single topic: 8 posts on hardware and the FreeBSD-based firewall solutions pfSense and OPNsense. Even after almost four years, some of these posts are still very high on the list of frequently visited pages. A lot has happened since then, though. About time that I get back to the topic! Here’s the list of the old posts:

Part 1 (discussing why you want to build your own router and how to assemble the APU2),
Part 2 (some Unix history explanation of what a serial console is),
Part 3 (demonstrating serial access to the APU and covering firmware update),
Part 4 (installing pfSense),
Part 5 (installing OPNsense instead)
Part 6 (Comparison of pfSense and OPNsense)
Part 7 (Advanced installation of OPNsense)
Part 8 (ZFS and jails)

I meant to revisit this topic again much sooner, but it didn’t work out. In 2018 I started to write a post that (among other text) contained the following paragraphs:

A lot has happened in the meantime. The new pfSense version that I mentioned before has left beta and is available regularly now. And with it comes one real boon: Complete ZFS support! OPNsense had two releases since 17.1 (the version that I discussed): 17.7 and 18.1. I had updated my box to 17.7 without any problems IIRC. After moving houses (and finally being online again after months!) I brushed the dust off my router and connected it. I decided to do a re-install this time and see if anything noteworthy had changed.

First I was a little disappointed to see that the USB3 issue has not yet been taken care off and still makes installing on the APU2 harder than it needs to be. But well, I had figured out how to do it previously, and still had my gear at hand to install using the internal USB2 connection. Second letdown: Still no trace of ZFS to be found. Looks like my priorities do not completely match the ones of the OPNsense team! 😉

But then the pleasant surprise: The 18.1.8 update brought experimental ZFS support! I had done a simple install this time as I had been a bit in a hurry, so I could not test it myself so far. However if I got things right, this means that the boot scripts are finally ZFS-capable and this will likely be supported in the 18.7 release. Chances are that there will not be installer support for ZFS, though. But we’ll see! Good things are going on and eventually we’ll get there.

I didn’t finish the article for reasons that escape me today, but when I wanted to return to the topic another year later, my hardware broke and so I had to postpone that again. My APU has since been fixed and even the old 16GB mSATA drive it originally came with replaced with a nice new one that has much more useful 512 GB of space – but I never found the time to even boot it up once since early 2020.

In late February 2021 I finally got around to install OPNsense on it again and thus get my own router back in business. I took notes but publishing this article was delayed for various reasons. Anyway, here we go! Let’s do a fully ZFS-based installation (which was not possible, yet in 2017).

Back in OPNsense land

When I first built my router, I had to be a bit creative to end up with an OPNsense installation that offered ZFS so that I could use a jails manager on it that depends on that filesystem. However it was only an additional pool for data. The actual system ran on UFS.

For a while now OPNsense does support booting from ZFS! I definitely want that, so let’s give it a try. I download a DVD image for OPNsense 21.1 and put that on my CD emulator device. Then I wire my APU up to my workstation via a serial cable and connect:

# cu -l /dev/cuaU0 -s 115200

Then I boot the machine up. As I’m using the DVD image, it’s configured for VGA mode. That means in my case I have to manually configure it for serial usage. This can be done conveniently in the loader in FreeBSD 12.2 and newer. However… OPNsense 21.1 is still based on version 12.1 where that option is not available, yet. Even though this is an older and in fact unsupported version, I cannot blame the project. They don’t use vanilla FreeBSD directly but are based on HardenedBSD instead, a close fork that is about hardening the code base.

While HardenedBSD has attracted enough attention and people to successfully form their own foundation, it’s still a small project with limited resources. And 12.1 is the only release from FreeBSD major version 12 that they support. If you like to support Open Source projects and have some spare money, consider donating. They are doing important work to move FreeBSD in the right direction!

So we have to do this the old way. Pressing ESC drops me at the loader prompt. Time to set some values and then boot:

set boot_serial=YES
set comconsole_speed=115200
set console=comconsole
boot

Alright! Kernel is booting up and printing all messages to the serial console. It shouldn’t take too long before… Whoo! What’s this?

Mounting from cd9660:/dev/iso9660/12_1_RELEASE_AMD64_CD failed with error 19.

Good lord, I remember this! OPNsense 17.1 (or was it 16.7?) had a problem booting up on some USB3 ports. You had to use a workaround to be able to boot up to the installer… I totally expected that this would have been fixed by now. It’s been four years! *sigh*

I quickly tried out installing from a usb memory stick – but ran into the same problem. So what now? Back in the day I opened up the case and connected an adapter to be able to use an internal USB2 connector. Of course I could do that again. Except – I have no idea whatsoever where I put that before moving houses… A little research doesn’t exactly leave me in a much better mood: As of OPNsense 21.1 they’ve apparently not added an easy way to install to ZFS, yet. At this point I’m ready to question if this project was a good idea or if I should simply do something else.

But this is not as bad as you might think. Fortunately there’s a very simple way of achieving it anyway: Just install vanilla FreeBSD and then use a conversion script. As I had just completed a series on PXE-booting (and the PXE server still at hand), I chose to go down that route. Take a look at that article (and the previous two) if you’re interested in the PXE booting details or don’t understand something that I do in the next section.

On the very day I’m finally writing this article though, the OPNsense team has announced switching the installer for the upcoming version 21.7, making installation to ZFS a regular installation option. So that’s one thing to look forward to if you like ZFS but don’t want to use the bootstrap script. 🙂

Installing FreeBSD

So I boot up my PXE server. It was prepared to serve FreeBSD 12.2. It is a very bad idea to try to bootstrap OPNsense on a version of FreeBSD that’s newer than the one the firewall OS is based on. So for this special case I add FreeBSD 12.1 support on my PXE server:

# mkdir /usr/local/www/pxe/bsd/fbsd/amd64/12.1-RELEASE
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.1-RELEASE/MANIFEST -o /usr/local/www/pxe/bsd/fbsd/amd64/12.1-RELEASE/MANIFEST
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.1-RELEASE/base.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.1-RELEASE/base.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.1-RELEASE/kernel.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.1-RELEASE/kernel.txz

By default, PXE-booting is disabled on the APU. So when turning on the device without any USB device attached to it, this is what I get on the serial console:

Press F10 key now for boot menu

Select boot device:

1. Payload [setup]
2. Payload [memtest]

So let’s see what options the setup has to offer:

Boot order - type letter to move device to top.

  d SATA
  c mSATA
  b SDCARD
  a USB
  e mPCIe1 SATA1 and SATA2
  f iPXE (disabled)


  r Restore boot order defaults
  n Network/PXE boot - Currently Disabled
  u USB boot - Currently Enabled
  t Serial console - Currently Enabled
  o UART C - Currently Enabled
  p UART D - Currently Enabled
  m Force mPCIe2 slot CLK (GPP3 PCIe) - Currently Disabled
  h EHCI0 controller - Currently Disabled
  l Core Performance Boost - Currently Enabled
  i Watchdog - Currently Disabled
  j SD 3.0 mode - Currently Disabled
  v IOMMU - Currently Disabled
  y PCIe power management features - Currently Disabled
  w Enable BIOS write protect - Currently Disabled
  x Exit setup without save
  s Save configuration and exit

Ok, using “n” I can enable PXE booting. Next time I get this message:

Press F10 key now for boot menu, N for PXE boot

Pressing N takes me to the iPXE boot menu. I quickly interfere to manually specify what to do instead of letting it automatically guess (and guess wrong). At the iPXE prompt I ask it to get an ip address and then chainload pxelinux like this:

iPXE> dhcp
Configuring (net0 00:0d:b9:42:67:64)...... ok
iPXE> chain pxelinux.0

There we are. As the mfsBSD image that I use is FreeBSD 12.2-based, I can set the console to serial in the loader (by pressing “5”) and then boot. Once the system is up, I login as root with the password mfsroot. Next is preparing the manifest and then starting the familiar FreeBSD installer:

# mkdir -p /usr/freebsd-dist
# fetch http://10.11.12.1/bsd/fbsd/amd64/12.1-RELEASE/MANIFEST -o /usr/freebsd-dist/MANIFEST
# bsdinstall

I want a minimal, bare-bones FreeBSD system and thus choose to not install any optional distsets. When asked about the installation source, I pick “other” and enter http://10.11.12.1/bsd/fbsd/amd64/12.1-RELEASE as that’s where my PXE server offers the required files. Now I can install FreeBSD using root-on-ZFS and everything as I like it.

When the installation is done, I reboot.

Converting FreeBSD to OPNsense

I enter setup again and disable network booting. As this is 12.1 again, I need to manually enable the system for use with the serial console. So when the loader menu comes up, I escape to the loader prompt and again use the same commands as above to set the variables to the right values and then boot.

After logging in as root, I download the conversion script:

# fetch --no-verify-peer https://raw.githubusercontent.com/opnsense/update/master/bootstrap/opnsense-bootstrap.sh

again, we’re using 12.1 which is the last version without without a certificate trust store in the base system. I could install ca_root_nss, but I’m choosing to just not validate the TLS cert this one time (the bootstrap process will do it correctly then).

All that’s left to do now is running the script:

# sh opnsense-bootstrap.sh
This utility will attempt to turn this installation into the latest
OPNsense 21.1 release.  All packages will be deleted, the base
system and kernel will be replaced, and if all went well the system
will automatically reboot.

Proceed with this action? [y/N]: y

It will then install packages and eventually the base system:

Fetching base-21.1.1-amd64.txz: ........................... done
Fetching kernel-21.1.1-amd64.txz: ....... done
!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!
! A critical upgrade is in progress. !
! Please do not turn off the system. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Installing kernel-21.1.1-amd64.txz... done
Installing base-21.1.1-amd64.txz... done
Please reboot.

Then it automatically reboots.

OPNing my senses

Settings for the serial console need to be entered again at the loader once more. Conversion worked like a charm: OPNsense boots up just fine. I’m starting over fresh here, so no configuration importer for me. But yes, I do want manual interface asignment.

For my box I do the following settings: No VLANs, manually configure: igb0 -> WAN, igb1 -> LAN, igb2 -> Opt

Once it’s done, I can connect from my workstation on the LAN to it by opening 192.168.1.1 in a browser. Login credentials are user root & password opnsense (yes, the conversion script wiped whatever root password you chose before).

Then I complete the wizard which mostly means changing the login password. Afterwards I go to System -> Firmware -> Settings and change the Firmware Flavor from the default OpenSSL to LibreSSL (don’t forget to save). Next is checking for updates and updating the system.

The Web UI is nice and all, but I’m a keyboard and terminal person. So for convenience I add a new user for me. To do so, I go to System -> Access -> Users. Obviously a username is required. I don’t want a password for that user, so I leave that blank and check the box “Generate a scrambled password”. My login shell of choice is /bin/tcsh (as zsh is unfortunately not in FreeBSD’s base system). Since I need a privileged user, I add “admins” to the group memberships. Then I paste in my public SSH key (from ~/.ssh/id_ed25519.pub on my workstation – if you’re using a different algorithm use the correct file name for that) and save.

The last thing that I do for settings is allowing my user to actually SSH into the box. So I’m going to System -> Settings -> Administration. There I check the box “Enable Secure Shell” and set “Listen Interfaces” to “LAN”. I’m also allowing passwordless sudo for “wheel,admins”. After I saved the settings, let’s see if I can connect to the box via SSH and become root:

% ssh 192.168.1.1
Enter passphrase for key '/home/kraileth/.ssh/id_ed25519':
----------------------------------------------
|      Hello, this is OPNsense 21.1          |         @@@@@@@@@@@@@@@
|                                            |        @@@@         @@@@
| Website:      https://opnsense.org/        |         @@@\\\   ///@@@
| Handbook:     https://docs.opnsense.org/   |       ))))))))   ((((((((
| Forums:       https://forum.opnsense.org/  |         @@@///   \\\@@@
| Code:         https://github.com/opnsense  |        @@@@         @@@@
| Twitter:      https://twitter.com/opnsense |         @@@@@@@@@@@@@@@
----------------------------------------------

% sudo -i
*** OPNsense.localdomain: OPNsense 21.1.1 (amd64/LibreSSL) ***

[...]

  0) Logout                              7) Ping host
  1) Assign interfaces                   8) Shell
  2) Set interface IP address            9) pfTop
  3) Reset the root password            10) Firewall log
  4) Reset to factory defaults          11) Reload all services
  5) Power off system                   12) Update from console
  6) Reboot system                      13) Restore a backup

Enter an option:

There we go. Now it’s time to configure my firewall settings. As this is a topic of it’s own, I’m going to skip this here.

Conclusion

Just like 4 years ago, there are still some hurdles to overcome to install OPNsense on my router (especially the USB3 issue). There’s also the fact that it’s based on FreeBSD 12.1 instead of 12.2 which is not ideal. But again: I’m not really complaining about that; HardenedBSD chose to concentrate on 13.0 and taking the resources of their small team into account, this was a perfectly reasonable decision. But regarding things like base system certificates and especially console selection in the loader… Well, you adapt to nice new features incredibly fast, making things you had to do before that (and did for years!) a bit of a nuisance. 😉

I started with OPNsense 17.1, I think and did a lot of point release updates as well as a couple of major release upgrades before my hardware went bad during the 18.7 or early 19.1 life cycle, I think. It had never failed me. Being able to use root-on-ZFS today is definitely nice progress. Since I installed the system in early March, a couple of updates came out. It’s great to see how reliably everything still works. So I’d say that I’m back on track – and I’ll definitely have a couple of things that I want to achieve (and write about) this time. Stay tuned!

Women… in tech?! (And elsewhere)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/women_in_tech.gmi

This is kind of a sensible topic, I know. But being a person used to stand up for his beliefs, I take the risk of being shouted at. I’m well aware that I’m not going to make friends in either of the two “camps” that usually fight each other over the topic. As always, I’m more interested in a balanced view – which of course means to criticize all the various ideologies involved.

Why do I write this article?

In my previous post I wrote about the new campaign to cancel Richard Stallman who has returned to the Free Software Foundation’s board of directors. It involved all the common drama about him being “sexist”, “transphobic” and so on.

Quite some interesting things happened since I wrote said article: The support letter for RMS ended up with more than double the amount of people signing it compared to the original open letter pressing for his removal.

I’ve read a very interesting article written by two women who also chose to support RMS in this case. If you care about the topic, have a look at what Hannah Wolfman-Jones wrote about a year ago (including a statement by Nadine Strossen). Also a more recent support article has been published by Leah Rowe (a transgender person).

And it even seems like the FSF is not going to give in this time! Thus it looks like this campaign failed (let’s hope that will happen more often in the future).

Something less important also happened; I’m only going to mention it here because it’s the direct reason for me to write this article. A reader commented on the RMS article. I’ll cite the part here that made me promise a longer statement than would fit into another comment:

Thirdly I’ll add here it’s insulting as Hell so many of you only pretend to care about Neurodiversity when a man is accused of and or being a creep, support of woman with ASD being harassed by men is never even considered.

I do not “pretend to care” about Neurodiversity. I do care about life and about society. This involves a lot of things including what is sometimes labeled Neurodiversity. But there is a reason why I put it all that general: I don’t support adding so much more value to a single aspect of the great whole. People who only care about particularities often think that I’m hostile towards their concerns. Usually I’m not. I just refuse to neglect other important aspects for the (alleged) benefit of another.

Do I care for women?

This is a very strange question, but nevertheless an interesting one. Not so much because of what my answer tells about me. More so because of what it tells about a person who would seriously ask it. Why? Well, if it’s not a rhetorical question, the person asking at least thinks that some form of “no” could be the answer. Which means that he or she is willing to believe that I might suffer from a severe mental illness.

I have a mother. I have a wife. And in fact I have daughters. How on earth could I not care about women? But let’s say somebody doesn’t have the previous information. What scenarios are there where I could answer “no” as a sane person? I could be a monk in a monastery who maybe would not have contact with women ever again in his life. This is a pretty extreme example already (and some people would doubt the sanity of anyone making such a decision but that’s not the point). I’d still adore mother Mary and it would be my religious duty to pray for other people (where limiting that to other men would not make any sense at all). Or maybe I’m a hermit, sick of it all, going to live in the woods by myself. Another rather extreme example. Perhaps I really wouldn’t care about women. But certainly not specifically! I wouldn’t care about them because I stopped caring for anybody.

Think about less radical examples. If you don’t flee society completely, women are almost guaranteed to be part of your life at least indirectly. The woman at the bank or on the counter in the store? Even if you are a pretty selfish person you won’t care less for them than you do for anybody else around you that you have no closer relation to. I tried to find one but could not come up with a somewhat plausible case where anybody would specifically not care for women and I wouldn’t question this person’s sanity (like with e.g. some ultra-orthodox Jews. It’s simply mind-boggling to find out about their views and forms of life!).

Alright, I do care. But what does that mean? I grew up in a society that was very much hostile to women in many regards – but people didn’t see that (not just men, a lot of women didn’t see or didn’t want to see it either!). As a child I chased the ideas of chivalry that I knew from books because I believed in that. I admired women as they seemed to be free from a lot of typical male… let’s say: defects. I did not understand that they have their own. 😉

Nailing my colors to the mast

As a child and later as a young man I stood up for women e.g. when dirty jokes were told in all male groups. I was called names for that (e.g. “gelding” and such). It was difficult and my protests and actions did not have any actual effect. But I at least tried, eh? The fact that I never dated a girl (until I was a bit older and was finally thinking about starting a family) made some people suspect that I was secretly gay. Which of course made matters worse.

While I cannot deny that I did feel attracted to girls, my beliefs in what partnership should be set me apart from almost all males in my peer group. Not being interested in some cheap quick pleasure does have one very positive side though: It frees you from the requirements of the “hunter and prey” game, like constantly having to think about what lie you should tell a girl next to eventually bed her. So I was able to simply be honest when talking to women. And this allowed for some very interesting discussions that I wouldn’t want to have missed.

My observation is that many woman have a good sense for a certain kind of danger. When they feel safe, they behave differently and they also speak differently. Knowing this I learned to fully accept that there have to be women-only places (and that there’s no need to feel “excluded” for not being admitted). There simply is no male equivalent to this (which is perfectly understandable if you think about it).

When I still was in school, I participated in one compulsory optional subject as the only boy in class for a while (a month or so later another boy switched classes and came into the course). I found it a little strange (mostly in the sense of “unusual” but nothing more) for the first few days, but I soon basically forgot about it. Much later I have been in a reversed situation: For a while there was only one girl in an otherwise all male class. She understood the subjects well, got good marks in written tests – but was admittedly a bit shy in class. She would definitely not participate when fellow students got passionate (and thus a bit louder) about several topics. It probably would have been easier for her in a more balanced class.

Culture

The biggest problem that I saw and still see is popular culture. What picture of women is being conveyed? I mentioned jokes. Let me present one pathetic example that I still remember:

Why do women have legs? – So they don’t leave a slime tail on the floor!

Ha ha, very funny, isn’t it?

Here’s another example, this time from music: A song called “Polonäse Blankenese” was popular in my country when I was a child. The artist, Gottlieb Wendehals, was more of a comedian. You have to take that into account, but still. Here’s the translation of a piece of the chorus:

We’re setting off, taking very big steps
And Erwin from behind touches Heidi’s… shoulder
That lifts the spirits, delight emerges
And that’s for all to see now

What’s the deal? Well, the verses rhyme in German – except for “shoulder”. What would obviously rhyme with the German word for steps however is – tits.

While you can dismiss such things as harmless jokes (and I wouldn’t encourage anyone going hysteric over it), if something like this was very popular both on TV and folk festivals and such, it certainly helps characterize the spirit of that time.

Fast forward a couple of years. In the late 90’s we had the Bloodhound Gang and their album “Hooray for Boobies”. I’m well aware that – again – it’s meant to be funny, but also a little insensitive perhaps? There were songs like their “Three Point One Four” (didn’t get the word play with that title back then) played on parties or even on the radio. This brought us great lyrics like “I need to find a – new vagina; Any kind ‘o – new vagina!”. I’m not saying that toilet humor and the like needs to die, but I do question if it really needs to have a place in the mainstream where it’s hard to evade?

And as we all know, it didn’t get better but in fact much worse. It didn’t stop with (unconscious as I’d claim) “humor”. Today we have certain “Hip Hop” bands for example who completely objectify women and entirely reduce them to their genitals. While Wendehals in the early 1980s wouldn’t actually say “Tits” and it was “perky” enough to hint it, there’s no lack of way more abasing words in popular culture today.

A fellow metal head once played a “fun” song on his phone for all of us to laugh about. The point was that someone took a “Hip Hop” song’s very misogynous lyrics and used them in another song of a genre where such a thing is… not what you’d expect. While I actually think the performance is not such a bad idea at all (because thanks to the grotesque change in genre, people who got used to what “Hip Hop” is like might think again). I didn’t feel well in that situation, though – because there was a girl with us. Obviously my classmate didn’t even think that there might be a problem with lyrics (translated) about “what the cunts really want”. I asked her later what she thought about that. Her reply was: She can laugh about that, you simply have to grow a bit of a thick skin – there’s no point in being offended!

While she was right of course, I don’t think it really has to be that way. In that scene nobody intended to hurt anybody after all. It’s simply mindlessness, not bad will.

There are so many examples from everyday life one could write about. I do not agree with each and every scandal that certain people call “sexist”. But it’s not like there’s no problem at all. There is. Understanding that there is in fact a whole class of problems that many of us don’t even notice is a very good first step. The second would be helping people who don’t want to remain ignorant see them. Then we could start talking about possible ways out of this mess.

Tech

Especially in tech some men seem to think that this is their playground and that women should simply do something else. That’s a somewhat strange position to start with. But does anybody really think “fighting” (in the classical sense) that way of thinking can do any good? Attacking somebody forces that person to go into defense mode. Doing that is a lousy tactic when you want to change someone’s mind!

There are more forms of attacking than just telling somebody very frankly that he’s an idiot. One example is dragging women into tech to fulfill some kind of “quota”. When a woman gets the job not because of her skills but because being a woman that’s a very problematic situation. There are men who will hate her for this and the woman will suffer from it. She might also struggle with the job that she did not get because she was well fit for due to her education. What a great combination!

My wife doesn’t like tech. She’s annoyed by all those “outreach” programs and everything that – according to her – tries to lure more women into IT. A lot of men also loathe these programs because they feel excluded (something they are not used to!). I think that looking at this slightly differently would be helpful.

Most people have at least heard that the female and the male brain work differently. While we’re all individuals and there are always those that think / feel / work differently than their peers, in general there’s a “male approach” and a “female approach” to things. And no, the female approach is not simply the same thing in pink! I’m all for separating boys and girls in certain school subjects and teaching them differently. This will help both unlock their potential as much as possible. There is no “one size fits all”!

Referring to this for example when justifying the existence of women’s courses might meet much less resistance from men: There’s a logical reason for it after all (and not just obscure “feelings” that not too few men have problems taking serious)! Let’s forget about the fact that women often tend to feel uneasy when they are in a mostly male environment – at least for a moment. We can fix the problem without waving the red rag and making the bull mad.

History

This article is becoming too long already. I intended to write the main part of this article about history: The Germanic, Slavic and Celtic culture and the position of women in it. About how Judaism (not monotheism! The first known monotheistic religion of Jatin in Ancient Egypt – thanks to the damnatio memoriae of Amenhotep IV. better known to us today in its Greek form of Aton or Aten – did not devalue women!) planted a seed that fanatical Christian monks who identified women with “sin” helped grow. This led to prosecution of wise women and eradicated the old rights that they once had.

I thought to give the example how even the term “woman” changed in my native language: It used to be “Weib” (still “weiblich” means simply “female”). In Middle High German “wîb” actually meant woman or wife. Today it’s negatively connoted, more like hag. Today the neutral term for woman is “Frau”. This is derived from “frouwe” which originally addressed a noble woman. If you want to translate lady today, you use “Dame” – so we had to resort to French because the Germanic words for women were devalued over time… (As you can imagine, no such thing happened to the word for “man”.)

And I wanted to write about how I (still in school) stopped admiring women as the better humans when I realized that some are perfectly capable of acting in just as malicious ways: On a school trip some girls conspired against another girl, treating her cruelly enough that she had to quit the trip! There are good and bad people of both genders – and there’s a lot of levels of gray with most women just as with most men. Women are special and I appreciate that a lot. They certainly are not inferior to men but neither are they morally superior in general.

If you think that the previous sentence is wrong, I recommend doing a broader research on the topic. At least read a book about Catherine the Great. Study the conflict between Mary Stewart and Elizabeth I. Get familiar with the epoch of the Catholic church called “Saeculum obscurum” during which weak men were popes and it is said that they were controlled by their more clever fancy women. The term “pornocracy” (as in Rule of Harlots) was coined for this. Or take Countess Báthory into consideration… Life is complex and there have been a lot of strange things going on and continue to do so. Never think that a simplistic view of something will carry the whole truth!

Can we change our ways?

There are people – mostly men – who refuse to see a problem. It’s useless to try to force them into abandoning old ways. Let’s concentrate on raising awareness for the problem with people who are open to see it. Then let’s discuss ideas and possible solutions.

I am deeply convinced that any undue force will actually thwart the undertaking. Forget banning words, getting into reverse witch-hunting and so on. I’m not interested in slightly brightening up the face of the building that is today’s society. What I’d rather look for is the cure. And that can most likely only be found in a new attempt of accepting each other. From that mutual acceptance respect can grow (I’m not talking about “respect” here, the contemporary empty phrase worn-down beyond recognition!).

When my grandparents from the mother’s side passed, I came by an old wooden decoration plate. Its translation reads: “It’s better to talk to each other than to keep silent against each other”. There were conflicts within their marriage for sure, but they remained together “in good times and in bad times” and overcame them. This is the way, both in a relationship and in society in general.

Can those of us who at least have some good will please try to get along despite our differences in opinion on particular issues? The struggle between sexes is entirely pointless – what “victory condition” would either “side” define? Let’s try to relax towards both men and women who think that they have to participate in that struggle. If more of us start looking for understanding each other, for kindness and re-unification to advance society as a whole together – then the people who preach hate, hostility and division will lose. In addition to our efforts it will only take time.

Cancelling Richard Stallman?

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/cancelling_rms.gmi

If you have any interest in FLOSS (Free and Libre Open-Source Software), you know who Richard Matthew Stallman (RMS) is: As the founder of both GNU project and the Free Software Foundation (FSF), he’s quite an icon to many. In 2019, a scandal around deliberate (?) misunderstanding of what he said regarding one detail of the Epstein affair ultimately lead to him stepping down as president of the FSF. Now in March 2021 he returned to the organization’s board of directors. The latter fact was reason enough for quite some people to start a turmoil again, pressing for his second removal with an open letter.

Now personally I don’t like RMS much. I believe in permissive licenses and prefer those over copyleft in general and strongly over the GPL license family that Stallman stands for like no other person. I’m a happy Vi user and think that Emacs (RMS’s editor) is a great example for what software should not be like. I’ve also regularly opposed false claims of Stallman’s many fans and their very pessimistic view on important topics like freedom and life in general. In fact I’ve used neologisms like Stallmanism and Stallmanites to describe the indiscriminate ideology of Stallman and his most pig-headed followers.

Today I’ve signed another Open Letter supporting RMS and I’m even writing this article. How come?

Cancel culture

Let me repeat: I do not particularly like the person RMS nor do I uncritically approve of what he stands for. On the contrary. But as an Open Source enthusiast and advocate who has tried to argue against his positions I do even less approve of how a mob of phony hypocrites is acting against him. Gesture politics is wrong and actually harmful. Let’s talk about real problems in this world and not publicly slay giants of the Free Software movement by basically backstabbing them!

The phenomenon of what is often called “cancel culture” these days is one brutal form of contemporary witch-hunt. It is deeply anti liberal and anti free speech. Real people are “convicted” not by a judge after at least being able to plead innocent and have their case examined duly. No, it’s a gang with a certain political agenda that decides someone is guilty of whatever and will just insist on action to be taken on their judgement. This is much, much more dangerous than most neutral observers think. If such a campaign is successful it means that de facto there is a new power along the actual written law – and it may even take precedence over it!

Let’s think about this for a moment. Maybe you’ve been a lawful citizen all your life. You are an esteemed member of society and maybe even earned honors and awards. But then all of the sudden somebody points at you and shouts: “You did XYZ thirty years ago!” While that was nothing special back in the day and was (and is!) perfectly legal, more and more people join in, screaming at you that this is a “disgrace” and completely “inexcusable”… How do you defend against such a campaign?

• You could explain that it’s not an illegal thing. Then the mob will eat you raw, yelling that this clearly shows “how you have not even learned a thing”!
• You could state that you are deeply sorry and honestly wish you had acted differently. But as your actions are by the mob’s definition “inexcusable”, you’re done for anyway.
• Or you could try a combination of both. Yet you’ll still drown in the public wrath directed at you.

Plead guilty or innocent before the judgement of the campaigners – it doesn’t make too much of a difference. And all of that while you are still very much spotless and respectable by law! This violates a number of fundamental legal principles of constitutional states:

• Nulla poena sine lege (“No punishment without a law”): If no rule exists that prohibits something, you cannot be punished for it.
• Nulla poena sine lege certa (“No punishment without precise law”): You cannot be convicted on base of a law that describes “somewhat similar” deeds.
• Nulla poena sine lege praevia (“No punishment without previous law”): If you did something before it was declared a crime, you cannot be punished. Ex postfacto laws are invalid!
• And more…

The effects of “cancel culture” that we see today are a true nightmare for any rational thinking person. You can be fried decades from now for something you do today without any bad will at all! Also it means turning away from the Christian principle of forgiveness for repentant offenders and thus shakes the very foundation of Western society as we know it. Welcome, friends, to our Brave New… τυραννίς! (I’m using the Greek word Tyrannis on purpose here, because I refer to an ancient Greek Tyrannos here and not modern notions of “Tyranny” and “Tyrant” which is a judgmental and negatively connoted term.)

Context!

Campaigns like the aforementioned against RMS work according their own twisted logic – and that only really applies if you can make people so upset that they are willing to ignore each and every context.

I’m a sysadmin; if you hear me talking about “killing all children” that does not make me a misanthropist or a violent person! Now I can hear you say: Fine, but there are things that are despicable no matter the context! Beg your pardon, but think again, please. Everything is subject to context.

Even though most people would agree to the statement “killing is wrong”, let me ask you if you really think your local butcher is a criminal (no food debate, please. I’ve been a vegetarian myself for close to a decade, but I am a liberal being accepting other people’s different thinking on such a matter)? It actually makes much of a difference if you think that “Thou shalt not kill” is one of the 10 commandments – or if you take a more precise translation of “Thou shalt not murder“!

Or for another example: Could there really be any context that would make the statement “rape should not be punished” anything but loathsome? Yes, of course! It can for example be a premiss in thinking about the consequences of such a stance. And that would be a perfectly legitimate thing: Unbiased thinking about something. There can be value in thinking over even the seemingly most absurd premisses like that.

There are two very important pieces of general context that the campaigners misappropriate:

• What was the general public feeling towards XYZ at that point in time? Was it really such a huge deal back then? If not: How can you dare to not only dictate your moral values to others but even demand different behavior when what you dislike today was still very common?
• RMS is autistic. And not even the very light form of it. People’s feelings are hard (maybe even impossible) for him to really understand.

Stallman is for example convinced that our planet is over-populated. When he learned that a person he had email contact with was going to be a father soon, he seriously expressed his sympathy – obviously not knowing that this is usually a joyful event for people…

Isn’t taking RMS’ autism into account a form of “abelism” by the campaigners? 🤪

Blame game

But let’s take a quick look of what people are accusing him. Here’s the beginning of the open letter with some comments from me:

Richard M. Stallman, frequently known as RMS, has been a dangerous force in the free software community for a long time. He has shown himself to be misogynist, ableist, and transphobic, among other serious accusations of impropriety.

Oh, wow. That’s a fairly “standard” set of charges. There’s nothing concrete here, just pretty weird claims made by people who love to hate opinions different from theirs…

These sorts of beliefs have no place in the free software, digital rights, and tech communities.

Sorry, but no. You presume to define something (in a grotesque way with no room for discussion about your definitions!), incriminate somebody (thus harming his reputation), hand down a verdict (without the need to even hear the defendant, because why should you?) and then press to enforce the judgment (ASAP of course). This is how things go in a dictatorship. We’re getting there, mind you, but are not quite there just yet. Get a life, stop being such ignorant jerks and prepare to have your creed challenged.

There’s more in the letter, but it all boils down to making more accusations and impudent demands. I don’t really want to waste any more time on it. Let’s hope that this is where such a campaign can finally be stopped.

If we could all return to discussing in a civilized manner again, that would be a giant step forward.

Re-learning to type… again! (From Neo to Bone)

Summary

After using an ergonomical keyboard layout for more than half a decade, I felt that I might try something new. My old layout (Neo) works great, but after learning of a newer variant (Bone), I wanted to try it out – and am fascinated by what the human brain is capable of doing.

Article

[New to Gemini? Have a look at my Gemini FAQ.]

The actual post can be found here (if you are using a native Gemini client):

gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/relearning_to_type.gmi

If you want to use a webproxy to read it from your web browser, follow this Link.

Multi-OS PXE-booting from FreeBSD 12: Linux, illumos and more (pt. 4)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/multi-os_pxe-booting_from_fbsd_pt4.gmi

Post 1 of this mini series is about what lead me to do this in the first place, features a little excursion for people new to PXE and details the setup of a FreeBSD router.
Post 2 discusses setting up the required daemons for DHCP, TFTP and HTTP / FTP. Each component that is not in FreeBSD’s base system is briefly discussed and two options to pick from are covered.
Post 3 covers the NBP (pxelinux), boot menu configuration and adding all the major BSD operating systems (with the exception of DragonFly that I could not get to work) as PXE-boot options.

In this post we are going to add some Linux distributions to the list as well as illumos and some other Open Source operating systems. If you are not familiar with pxelinux and configuring its menu, please see the previous post for more information. You will also find an explanation on going with flat menus if you prefer that – in this article I’ll make use of sub-menus.

Menu preparations

First is a bit of preparation work. I’m going to add three more sub-menus to the main menu – follow along with any or all that are relevant to you. On your PXE boot server, edit the file main configuration file for pxelinux:

# vi /usr/local/tftpboot/pxelinux.cfg/default

Append the following blocks:

LABEL linux-distros
        MENU LABEL Linux Distributions
        KERNEL vesamenu.c32
        APPEND pxelinux.cfg/linux

LABEL illumos-distros
        MENU LABEL illumos Distributions
        KERNEL vesamenu.c32
        APPEND pxelinux.cfg/illumos

LABEL other-oses
        MENU LABEL Other Operating Systems
        KERNEL vesamenu.c32
        APPEND pxelinux.cfg/other

Next is creating the configuration files for the sub-menus we just referenced. We’ll start with Linux:

# vi /usr/local/tftpboot/pxelinux.cfg/linux

Let’s put the menu title and an option to go back to the main menu in there:

MENU TITLE PXE Boot Menu (Linux)

LABEL main-menu
        MENU LABEL Main Menu
        KERNEL vesamenu.c32
        APPEND pxelinux.cfg/default

Now the same thing for illumos:

# vi /usr/local/tftpboot/pxelinux.cfg/illumos

Insert this text:

MENU TITLE PXE Boot Menu (illumos)

LABEL main-menu
        MENU LABEL Main Menu
        KERNEL vesamenu.c32
        APPEND pxelinux.cfg/default

And finally for the other systems:

# vi /usr/local/tftpboot/pxelinux.cfg/other

Again put the default stuff in there:

MENU TITLE PXE Boot Menu (Other)

LABEL main-menu
        MENU LABEL Main Menu
        KERNEL vesamenu.c32
        APPEND pxelinux.cfg/default

Alright, everything is prepared. So we can add some actual boot options for our PXE boot server!

Ubuntu

Making Ubuntu available over PXE is not very hard to do. We first need the (very recently released) ISO. It’s a big file and thus it makes sense to check the downloaded image for transmission errors – for that reason we’re getting the checksum file, too. The ISO is needed for the installation, so we’re going to make it available over HTTP by putting it in the right location:

# mkdir -p /usr/local/www/pxe/linux/ubuntu
# fetch https://releases.ubuntu.com/20.04/ubuntu-20.04.2-live-server-amd64.iso -o /usr/local/www/pxe/linux/ubuntu/ubuntu-20.04.2-live-server-amd64.iso
# fetch https://releases.ubuntu.com/20.04/SHA256SUMS -o /tmp/SHA256SUMS
# grep -c `sha256 /usr/local/www/pxe/linux/ubuntu/ubuntu-20.04.2-live-server-amd64.iso | awk '{ print $NF }'` /tmp/SHA256SUMS
# rm /tmp/SHA256SUMS

Make sure that the grep command line returns 1. If it doesn’t, remove the ISO and checksum file and re-download. We need the kernel and the init ramdisk available via TFTP, so we first extract the ISO to a temporary location. I found 7zip a good fit for extracting, so install it if you don’t have it on your system. Now copy off what we need and remove the directory again:

# 7z x /usr/local/www/pxe/linux/ubuntu/ubuntu-20.04.2-live-server-amd64.iso -o/tmp/ubuntu
# mkdir -p /usr/local/tftpboot/linux/ubuntu
# cp /tmp/ubuntu/casper/vmlinuz /usr/local/tftpboot/linux/ubuntu
# cp /tmp/ubuntu/casper/initrd /usr/local/tftpboot/linux/ubuntu
# rm -r /tmp/ubuntu

With all the system data in place we only need to add a boot menu entry for it:

# vi /usr/local/tftpboot/pxelinux.cfg/linux

Add the following to the file and save:

LABEL ubuntu-server-pxe-install
        MENU LABEL Install Ubuntu 20.04 (PXE)
        KERNEL linux/ubuntu/vmlinuz
        INITRD linux/ubuntu/initrd
        APPEND root=/dev/ram0 ramdisk_size=1500000 ip=dhcp url=http://10.11.12.1/linux/ubuntu/ubuntu-20.04.2-live-server-amd64.iso

Since pxelinux supports fetching via HTTP, you can also serve both the kernel and initrd via that protocol instead. If you want to do that, simply move the files to the other location:

# mv /usr/local/tftpboot/linux/ubuntu/vmlinuz /usr/local/tftpboot/linux/ubuntu/initrd /usr/local/www/pxe/linux/ubuntu

Then edit the menu definition again:

# vi /usr/local/tftpboot/pxelinux.cfg/linux

And adjust the menu definition like this:

        KERNEL http://10.11.12.1/linux/ubuntu/vmlinuz
        INITRD http://10.11.12.1/linux/ubuntu/initrd

And that’s it. Once you’ve restarted inetd, Ubuntu will be available as an option for PXE clients booting off of your server. It takes quite a while to push all the bits of the large ISO across the wire (’cause Ubuntu…) but it works.

AlmaLinux

What’s AlmaLinux some people may ask? Well, it’s one of the candidates for the succession of CentOS 8 after that was killed off by IBM. There are other candidates, but if you ask me, this one is the most promising right now. And since it’s available already (though in Beta currently), we’re not going to waste our time with dying CentOS but rather aim for the future of an enterprisy Linux distribution with Alma.

Obviously the first thing to do is getting it on our machine. I suggest getting the DVD which comes with all the packages available; if you know that you only do very basic installations, you may get the x86_64-minimal.iso one instead. Since it’s a pretty big ISO (8 GB!), checking for integrity is something I wouldn’t recommend skipping:

# mkdir -p /usr/local/www/pxe/linux/alma
# fetch https://repo.almalinux.org/almalinux/8.3-beta/isos/x86_64/AlmaLinux-8.3-beta-1-x86_64-dvd1.iso -o /usr/local/www/pxe/linux/alma/AlmaLinux-8.3-beta-1-x86_64-dvd1.iso
# fetch https://repo.almalinux.org/almalinux/8.3-beta/isos/x86_64/CHECKSUM -o /tmp/CHECKSUM
# grep -c `sha256 /usr/local/www/pxe/linux/alma/AlmaLinux-8.3-beta-1-x86_64-dvd1.iso | awk '{ print $NF }'` /tmp/CHECKSUM
# rm /tmp/CHECKSUM

If the grep command line does not return 1, delete both files and re-fetch, then run the check again. As soon as we have a good copy, we can extract the contents, making it available over HTTP. Permissions are wrong after extraction with all the directories missing the executable bit (rendering all their content inaccessible to the web server). So we fix this quickly, too:

# 7z x /usr/local/www/pxe/linux/alma/AlmaLinux-8.3-beta-1-x86_64-dvd1.iso -o/usr/local/www/pxe/linux/alma/sysroot
# find /usr/local/www/pxe/linux/alma -type d | xargs chmod 0755

The next step is to make kernel and init ramdisk available via TFTP:

# mkdir /usr/local/tftpboot/linux/alma
# cp /usr/local/www/pxe/linux/alma/sysroot/images/pxeboot/vmlinuz /usr/local/www/pxe/linux/alma/sysroot/images/pxeboot/initrd.img /usr/local/tftpboot/linux/alma

And finally we need to edit the Linux menu:

# vi /usr/local/tftpboot/pxelinux.cfg/linux

The definition to append looks like this:

LABEL alma-pxe-install
        MENU LABEL Install AlmaLinux 8.3 BETA (PXE)
        KERNEL linux/alma/vmlinuz
        INITRD linux/alma/initrd.img
        APPEND ip=dhcp inst.stage2=http://10.11.12.1/linux/alma/sysroot inst.xdriver=vesa

If you think that requiring X11 to install an OS is a reasonable choice and prefer the graphical installer, leave out the inst.xdriver=vesa option in the APPEND line. Also you can serve kernel and initrd via HTTP if you like. To do so, move the boot files:

# mv /usr/local/tftpboot/linux/alma/vmlinuz /usr/local/tftpboot/linux/alma/initrd.img /usr/local/www/pxe/linux/alma

Then edit the menu again:

# vi /usr/local/tftpboot/pxelinux.cfg/linux

And make the following changes:

        KERNEL http://10.11.12.1/linux/alma/vmlinuz
        INITRD http://10.11.12.1/linux/alma/initrd.img

Now your PXE server is ready to serve AlmaLinux.

Devuan Beowulf

Let’s add Devuan to the list next. Why Devuan and not Debian proper you ask? Well, because it has much cooler release names! Ok, of course not (the names are cooler, though!). Because of Init freedom.

While I don’t use Devuan (or Debian) myself, I’ve been supporting the fork since it started, sharing the first ISOs via Torrent and such. I very much dislike the “Bwahaha, you’ll never manage to fork XYZ!!!!11″ stance that some very loud people in the Linux community display. I felt ashamed for my fellow Open Source supporters when it happened to the MATE desktop and I did again when people dismissed Devuan like that. Let me emphasize: This ridiculing is not the Debian project’s fault. But the choice of opening their doors widely to Systemd while discriminating against everything else is. Hence: Devuan.

If you prefer Debian instead, there’s not too much to change: Substitute the directory name, grab Debian’s netboot archive instead and you’re good to go. First thing to do is to get the netboot image, extract it and copying kernel and initrd over:

# fetch https://pkgmaster.devuan.org/devuan/dists/beowulf/main/installer-amd64/current/images/netboot/netboot.tar.gz -o /tmp/netboot.tar.gz
# mkdir /tmp/devuan && tar -C /tmp/devuan -xvf /tmp/netboot.tar.gz
# mkdir /usr/local/tftpboot/linux/devuan
# cp /tmp/devuan/debian-installer/amd64/linux /tmp/devuan/debian-installer/amd64/initrd.gz /usr/local/tftpboot/linux/devuan
# rm -r /tmp/devuan /tmp/netboot.tar.gz

Then we have to edit the Linux boot menu configuration:

# vi /usr/local/tftpboot/pxelinux.cfg/linux

Simply add the following block to the file and save:

LABEL devuan-pxe-install
        MENU LABEL Install Devuan Beowulf (PXE)
        KERNEL linux/devuan/linux
        INITRD linux/devuan/initrd.gz
        APPEND vga=788

And yes, that’s already it. Debian does a pretty good job when it comes to PXE installing! If you prefer to serve the boot files via HTTP, just move the files:

# mkdir -p /usr/local/www/pxe/linux/devuan
# mv /usr/local/tftpboot/linux/devuan/linux /usr/local/tftpboot/linux/devuan/initrd.gz /usr/local/www/pxe/linux/devuan

Then edit the menu config again:

# vi /usr/local/tftpboot/pxelinux.cfg/linux

Now substitute the respective lines with these:

        KERNEL http://10.11.12.1/linux/devuan/linux
        INITRD http://10.11.12.1/linux/devuan/initrd.gz

Done. You can boot into the Devuan installer via PXE.

Alpine Linux 3.13

One last Linux example here is for a long-time favorite distribution of mine: Alpine Linux. It’s a very light-weight, security-oriented distribution based on Musl Libc. Alpine is well fit for embedded use and can run diskless and entirely from RAM – of course traditional installations are possible, too. Most people probably know it because it’s used frequently with Docker.

Adding Alpine to the mix is very simple. Let’s get and extract the netboot archive first:

# fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/releases/x86_64/alpine-netboot-3.13.1-x86_64.tar.gz -o /tmp/alpine-netboot-3.13.1-x86_64.tar.gz
# mkdir /tmp/alpine && tar -C /tmp/alpine -xvf /tmp/alpine-netboot-3.13.1-x86_64.tar.gz

Now we copy off kernel, initrd as well as the modloop:

# cp /tmp/alpine/boot/vmlinuz-lts /tmp/alpine/boot/initramfs-lts /usr/local/tftpboot/linux/alpine
# mkdir /usr/local/www/pxe/linux/alpine
# cp /tmp/alpine/boot/modloop-lts /usr/local/www/pxe/linux/alpine

Finally edit the Linux menu config:

# vi /usr/local/tftpboot/pxelinux.cfg/linux

Put the following in there and save:

LABEL alpine-pxe-install
        MENU DEFAULT
        MENU LABEL Alpine Linux 3.13 (PXE)
        KERNEL linux/alpine/vmlinuz-lts
        INITRD linux/alpine/initramfs-lts
        APPEND ip=dhcp modloop=http://10.11.12.1/linux/alpine/modloop-lts alpine_repo=http://dl-cdn.alpinelinux.org/alpine/edge/main

And that’s really it! If you wish to use HTTP only, move two files:

# mv /usr/local/tftpboot/linux/alpine/* /usr/local/www/pxe/linux/alpine

Then edit the menu again:

# vi /usr/local/tftpboot/pxelinux.cfg/linux

And change the following lines:

        KERNEL http://10.11.12.1/linux/alpine/vmlinuz-lts
        INITRD http://10.11.12.1/linux/alpine/initramfs-lts

That’s all, you’re good to go.

SmartOS

SmartOS is a gem unknown to many. It’s an illumos distribution (OpenSolaris-derivative) meant to operate fully in live-mode. That is: It uses the machine’s hard disk(s) only for storage – the OS is always booted via PXE. This means updating is as easy as booting a newer image! Due to its nature it’s really no wonder that it has the most advanced, stunningly simple PXE process of all the systems that I’ve covered so far. You just get the image file:

# mkdir -p /usr/local/www/pxe/illumos/smartos
# fetch https://us-east.manta.joyent.com/Joyent_Dev/public/SmartOS/smartos-latest-USB.img.gz -o /usr/local/www/pxe/illumos/smartos/smartos-latest-USB.img.gz

Then you edit the illumos menu config:

# vi /usr/local/tftpboot/pxelinux.cfg/illumos

Paste in the following block and save:

LABEL smartos-pxe
        MENU LABEL SmartOS (PXE)
        KERNEL memdisk
        INITRD http://10.11.12.1/illumos/smartos/smartos-latest-USB.img.gz
        APPEND harddisk raw

And that’s really it! There is no next step (except for enjoying your new system(s))!

Tribblix

Tribblix is a retro style illumos distribution that you might enjoy if you are a Solaris veteran. I did not succeed to get it working with pxelinux which seems to have trouble loading the illumos kernel. Fortunately, PXE-booting Tribblix using iPXE is officially supported. Yes, in a previous article I wrote that I decided to use pxelinux instead, but since there’s no other way (that I know of… please comment if you know how to get it work with just pxelinux!), we’re going down that route here.

The first thing to do is to make sure we’re able to chain-load iPXE. FreeBSD provides a package for it which we’re going to install. Then we make the required program available via http:

# pkg install -y ipxe
# mkdir -p /usr/local/www/pxe/other/ipxe
# cp /usr/local/share/ipxe/ipxe.pxe /usr/local/www/pxe/other/ipxe

Next is adding a menu item to select iPXE. So we need to edit the respective config file:

# vi /usr/local/tftpboot/pxelinux.cfg/illumos

Let’s add the following block and save the file:

LABEL tribblix-over-ipxe
        MENU LABEL Tribblix 0m24 (iPXE)
        KERNEL pxechn.c32
        APPEND http://10.11.12.1/other/ipxe/ipxe.pxe

This will be enough to chain-load iPXE. It’s not of too much use, yet, though. If you select it, it’ll request an IP address and use the PXE-related info it gets from the DHCP server to – load pxelinux. Press CTRL-B when it loads to break the automatic behavior and be dropped to a console prompt. However we can’t do much with it, yet, since we’re missing the OS data.

The location to use is a bit odd here. This is due to hard-coded paths in the image, I assume. At least I could not get it working using the scheme that I use for the other examples. Anyway, time to fetch the kernel and the ramdisk image:

# mkdir -p /usr/local/www/pxe/m24/platform/i86pc/kernel/amd64
# fetch http://pkgs.tribblix.org/m24/platform/i86pc/kernel/amd64/unix -o /usr/local/www/pxe/m24/platform/i86pc/kernel/amd64/unix
# fetch http://pkgs.tribblix.org/m24/platform/i86pc/boot_archive -o /usr/local/www/pxe/m24/platform/i86pc/boot_archive

Now that we have the data in place, we could manually boot up Tribblix. But there’s an easier way by creating a script which we can invoke and that simply does the rest. Let’s create it in the tftp directory (it’s small and using TFTP means we have less to type because we can omit the http://10.11.12.1/ part):

# vi /usr/local/tftpboot/tribblix.ipxe

Paste the following into the file and save:

#!ipxe
kernel http://10.11.12.1/m24/platform/i86pc/kernel/amd64/unix
initrd http://10.11.12.1/m24/platform/i86pc/boot_archive
boot

That’s it. Drop to iPXE’s prompt as described above and request an IP address; then simply chainload the script. Tribblix will boot to the installer:

iPXE> dhcp
Waiting for link-up on net0... ok
Configuring (net0 10:bf:48:df:e9:65)...... ok
iPXE> chain tribblix.pxe

Yes, this is somewhat less convenient than other options shown here. And in fact it can be avoided. The problem being that creating a more convenient solution is also a little bit more involved and simply beyond the scope of this article. If you’re interested in doing it, do some reading on your own. What you want to research is embedding the script into iPXE.

OmniOSce r151036

After struggeling with getting OmniOSce to PXE-boot to no avail for days I gave up and moved on to Tribblix. I wish I had done it the other way round; Peter Tribble has posted an article on his blog on how to do it with iPXE. And since I gave in and used it in my example above, I figured that I might as well return and give OmniOSce a try again. And really: Using iPXE instead of pure pxelinux – I got it working.

Caution, though: What is described here is a fully automated (unattended) installation. It will destroy any data on the target machine without asking. If that’s not what you want, stay away. I wish I knew a way to just invoke the installer instead, but I don’t.

Of course we need the operating system kernel and image as well as the zfs stream to install:

# mkdir -p /usr/local/www/pxe/illumos/omniosce/platform/i86pc/amd64 /usr/local/www/pxe/illumos/omniosce/platform/i86pc/kernel/amd64
# fetch https://downloads.omnios.org/media/r151036/omniosce-r151036m.unix -o /usr/local/www/pxe/illumos/omniosce/platform/i86pc/kernel/amd64/unix
# fetch https://downloads.omnios.org/media/r151036/omniosce-r151036m.miniroot.gz -o /usr/local/www/pxe/illumos/omniosce/platform/i86pc/amd64/omniosce-r151036m.miniroot.gz
# gunzip /usr/local/www/pxe/illumos/omniosce/platform/i86pc/amd64/omniosce-r151036m.miniroot.gz
# fetch https://downloads.omnios.org/media/r151036/omniosce-r151036.zfs.xz -o /usr/local/www/pxe/illumos/omniosce/platform/i86pc/amd64/omniosce-r151036.zfs.xz

Next is the installer configuration. Kayak will look for a file at the location told to (see below in the script) – the filename it tries first is named after the MAC address of the interface used for DHCP. In my case the MAC is 10:bf:48:df:e9:65, so the file matching only this single machine would be called 10BF48DFE965 (use uppercase for the Hex values above 9 resembled by the letter symbols).

If it cannot find that file it will look for a file with the name of the first 11 digits, i.e. 10BF48DFE96. That would also match MAC addresses like 10BF48DFE961, 10BF48DFE96A, 10BF48DFE96E and the like. If there’s no such file, it will try to get one named after the first 10 digits and so on until it eventually tries “1” in my case (if your MAC starts with a different digit, you will have to use that).

I’m lazy here and use a single-digit name (and thus choose to play with fire since this configuration will potentially nuke the previous installation of any machine having a network card with a MAC address that starts with 1 and would be PXE-booting OmniOS by accident!):

# vi /usr/local/www/pxe/kayak/1

I put the following in there (your machines are likely different from mine; refer to the OmniOSce documentation for more information about about the configuration script):

BuildRpool c3t0d0
RootPW '$5$JQkyMDvv$pPzEUsvP/rLwURyrpwz5i1SfVqx2QiEoIdDA9ZrG271'
SetRootPW
SetHostname omnios
SetTimezone UTC
EnableDNS example.com
SetDNS 1.1.1.1 80.80.80.80
Postboot '/sbin/ipadm create-if rge0'
Postboot '/sbin/ipadm create-addr -T dhcp rge0/v4'
SendInstallLog

Now I create a menu entry for pxelinux by editing the configuration file:

# vi /usr/local/tftpboot/pxelinux.cfg/illumos

I add the following block and save:

LABEL omniosce-auto-over-ipxe
        MENU LABEL OmniOSce r151036 AUTOMATED INSTALLATION [dangerous!] (iPXE)
        KERNEL pxechn.c32
        APPEND http://10.11.12.1/other/ipxe/ipxe.pxe

Finally I create the script to “chainload” from iPXE:

# vi /usr/local/tftpboot/omniosce.ipxe

Here’s the content for that script:

!ipxe
kernel http://10.11.12.1/illumos/omniosce/platform/i86pc/kernel/amd64/unix -B install_media=http://10.11.12.1/illumos/omniosce/platform/i86pc/amd64/omniosce-r151036.zfs.xz,in
stall_config=http://10.11.12.1/illumos/omniosce
initrd http://10.11.12.1/illumos/omniosce/platform/i86pc/amd64/omniosce-r151036m.miniroot
boot

And that’s it. Like with Tribblix, use CTRL-B to get to an iPXE prompt, obtain an IP address and chainload the iPXE script – in this case the one for OmniOSce. If you did everything correct for your machine (depending on its MAC), kayak should erase your data and install OmniOSce on it instead.

FreeDOS 1.2

Why add FreeDOS? Well, why not? I’ve played with it quite a bit in my youth and happen to like it. It’s also simple to do and can in fact be useful (e.g. when you want to flash some firmware). As always, the first thing is to get and extract the archive. The zip file contains a virtual machine image, too, but we don’t need it and thus skip unzipping it:

# fetch http://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/distributions/1.2/FD12FULL.zip -o /tmp/FD12FULL.zip
# unzip -d /tmp -x '*.vmdk' /tmp/FD12FULL.zip

Next step is putting the extracted image into the right place, to correct permissions and to compress it to save a fair bit of space:

# mkdir -p /usr/local/www/pxe/other/fdos
# mv /tmp/FD12FULL.img /usr/local/www/pxe/other/fdos
# chmod 0644 /usr/local/www/pxe/other/fdos/FD12FULL.img.gz
# gzip -9 /usr/local/www/pxe/other/fdos/FD12FULL.img

Finally edit the “other” boot menu config file:

# vi /usr/local/tftpboot/pxelinux.cfg/other

And add the following block:

LABEL fdos-pxe-install
       MENU LABEL Install FreeDOS 1.2 (PXE)
       KERNEL memdisk
       INITRD http://10.11.12.1/other/fdos/FD12FULL.img.gz
       APPEND harddisk raw

Done. You should be able to PXE-boot into FreeDOS now.

Plan 9

Back to the Unixy stuff – and (disregarding Inferno) the one OS that is more Unixy that Unix itself: Plan 9! I’m going with the 9front distribution here; it’s basically an example, anyway. There are not that many people out there who will want to actually provision a lot of servers with Plan 9, right? (If you are one of those, get in touch with me! I’d be interested in what you’re doing with it.)

As always we first need to get the image and put it in place:

# mkdir /usr/local/www/pxe/other/9front
# fetch http://9front.org/iso/9front-8013.d9e940a768d1.amd64.iso.gz -o /usr/local/www/pxe/other/9front/9front-8013.d9e940a768d1.amd64.iso.gz

Then the menu configuration needs to be edited:

# vi /usr/local/tftpboot/pxelinux.cfg/other

Paste in the following block:

LABEL 9front-pxe-install
       MENU LABEL Install 9front (PXE)
       KERNEL memdisk
       INITRD http://10.11.12.1/other/9front/9front-8013.d9e940a768d1.amd64.iso.gz
       APPEND iso raw

And that’s all there is to it.

Redox 0.6

Redox is an experimental OS is about implementing a POSIX-like operating system written in Rust. It’s using a modern microkernel architecture and made amazingly quick progress after the project started. I’m including it here because I think it’s a very interesting thing to follow.

To add Redox to the mix, you have to download the ISO first:

# mkdir /usr/local/www/pxe/other/redox
# fetch https://gitlab.redox-os.org/redox-os/redox/-/jobs/31100/artifacts/raw/build/img/redox_0.6.0_livedisk.iso.gz -o /usr/local/www/pxe/other/redox/redox_0.6.0_livedisk.iso.gz

Next and final step is editing the the menu configuration:

# vi /usr/local/tftpboot/pxelinux.cfg/other

and adding a block for Redox’ menu item:

LABEL redox-pxe-install
       MENU LABEL Install Redox OS (PXE)
       KERNEL memdisk
       INITRD http://10.11.12.1/other/redox/redox_0.6.0_livedisk.iso.gz
       APPEND iso raw

The current version seems to have introduced a regression regarding my test hardware. It boots but after setting the resolution, the computer simply restarts. Version 0.4.0 could boot up to the graphical login, though (but I couldn’t login due to missing input drivers I assume).

What’s missing?

I would have liked to add OpenIndiana, Haiku, Minix and ReactOS as well.

With both Haiku and ReactOS I failed to get them even booting. It might be possible to do it, but I ran out of ideas to try quickly and because of a lack of time didn’t try harder.

For OpenIndiana I didn’t even find proper documentation. Since I’m not very knowledgable with illumos, anyway, I eventually settled on having three other distributions covered and didn’t really try it out.

Minix is a special case. According to the documentation, PXE-booting should be possible with a ramdisk image. The project doesn’t provide them for download (or I didn’t find them) but building one from source is documented. So I checked out the source and ran the script to build it – and it failed. I tried again on a 64-bit machine and it failed again. Looks like cross-building Minix on FreeBSD did once work (documentation mentions FreeBSD 10), but this no longer seems to be the case. Not having any Linux machines around to try it on, I decided to skip it.

Conclusion

The 4 articles of this mini series detail how you can setup a PXE server on FreeBSD that will serve multiple operating systems depending on the menu item chosen by the user. With all the examples of Linux, illumos and various other Open Source operating systems in this article, you have plenty of choice for experimenting in your home lab!

This does not mean that there are not many, many more options available for free. If you know how to add more options – why not write about it yourself? Be sure to let me know, I’ll happily place a link to your work.

And as always: Feedback, suggestions for improvements and such are very welcome. Have fun!

Multi-OS PXE-booting from FreeBSD 12: PXE menu and *BSD (pt. 3)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/multi-os_pxe-booting_from_fbsd_pt3.gmi

Post 1 of this mini series is about what lead me to do this in the first place, features a little excursion for people new to PXE and details the setup of a FreeBSD router.
Post 2 discusses setting up the required daemons for DHCP, TFTP and HTTP / FTP. Each component that is not in FreeBSD’s base system is briefly discussed and two options to pick from are covered.

At the end of part 2, the situation is as follows: A client in the 10.11.12.0/24 subnet attempting to PXE boot will get an IP address via DHCP and be told where to get the NPB (Network Bootstrap Program). It will then attempt to fetch it via TFTP. There’s just one problem: That is not there, yet! We’ll fix that in a minute. When we’re done here, you’ll have a fully working PXE server that offers multiple BSD operating systems to boot into (Linux and more are covered in part 4). This article focuses on BIOS (“legacy”) booting; if you want to boot EFI-only machines you’ll have to adapt the configuration examples given here to that. I also assume using HTTP here – if you opted for FTP you will have to adapt the commands used in the examples.

Network Bootstrap Program

There are a lot of NBPs available. Usually each operating system has its own which is tuned towards its specific duty: FreeBSD has one, OpenBSD has another and Linux has several. These are not ordinary programs; they need to cope with a very resource-constrained environment and cannot depend on any external libraries. While writing boot code is challenging enough, adding network booting capabilities doesn’t make things any easier. This is why most NBPs are as simple as possible.

As a result of that, the NBPs usually know how to boot exactly one operating system. Since we want to set up a multi-OS PXE server this is quite unfortunate for our use case. There are two ways to work around this problem:

1. Provide various NBPs and use DHCP to distinguish between clients
2. Use an NBP that supports a menu to select which one to boot next

As usual there’s pros and cons to both. Letting DHCP do the magic requires a much more complex DHCP configuration. It’s also much less flexible. The boot menu approach is simple and flexible, but more complicated if you are also interested in automation. I do like automation, but I decided in favor of using a boot menu for this article because it’s easier to follow. It is also a good achievement to build upon once you’re comfortable with DHCP and feel ready for advanced settings.

It is possible to use one NBP to fetch and execute another one. This process is known as chainloading. For some operating systems that is the best choice to add them to a menu system.

There’s three popular options that we have for an NBP which fits our needs:

1. GRUB
2. PXELINUX (from Syslinux) and
3. iPXE

GRUB and I never made friends. I used it for a while after switching from LILO only to ditch it for Syslinux when I learned of that. I have to use it on many systems, but when I have a choice, I choose something else. The iPXE project is very interesting. It’s the most advanced (but also most involved) of the three options. If you’re curious about just how far you can take PXE booting, at least have a look at it. For this article, we’ll go with PXELINUX.

Pxelinux

Pxelinux is available via packages on FreeBSD. It does pull in some dependencies that I don’t want on my system however. For that reason we’re going to fetch the package instead of installing it. Then we extract it manually:

# pkg fetch -y syslinux
# mkdir /tmp/syslinux
# tar -C /tmp/syslinux -xvf /var/cache/pkg/syslinux-6.03.txz

Now we can cherry-pick the required files:

# cp /tmp/syslinux/usr/local/share/syslinux/bios/core/lpxelinux.0 /usr/local/tftpboot/pxelinux.0
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/elflink/ldlinux/ldlinux.c32  /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/menu/vesamenu.c32 /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/lib/libcom32.c32 /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/libutil/libutil.c32 /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/modules/pxechn.c32 /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/memdisk/memdisk /usr/local/tftpboot/
# rm -r /tmp/syslinux

The first one is the modular NBP itself. It requires some modules – the c32 files. The pxechn and memdisk modules are optional – they are required for some of the operating systems examples here but not all. You can leave them out if you don’t need them. Restart the inetd service now and you will be able to PXE boot to the menu:

# service inetd restart

Keep in mind: Restart the inetd service whenever you added a file to or edited any in the tftpboot directory!

Tip 1: You can use make use of gzipped files as Pxelinux supports that. This way you can decrease loading times by serving smaller images over the net.

Tip 2: I’m using gzip in my examples but if you really want to fight for the last byte, use zopfli instead. It’s a compression program that produces gzip-compatible output but takes much more processor time to create optimized archives. As decompression time is unaffected it’s a price you have to pay only once. Consider using it if you want the best results.

Submenus

Pxelinux is hard-coded to load pxelinux.cfg/default via TFTP and use that as its configuration. If you plan to only use few of the OS examples shown here, that config is sufficient as you can put everything into there. Once you feel that your menu is becoming overly crowded, you can turn it into a main menu and make use of submenus to group things as I do it here. This works by putting the various OS entries in different config files.

If you don’t want submenus, skip the next step and put all the menu entries that go into something other than pxelinux.cfg/default in my examples right into that file instead – and leave out the reference back to the main menu since they don’t make any sense if you’re using a flat menu anyway.

In the previous post we already created the file /usr/local/tftpboot/pxelinux.cfg/default. Append the following to it to create a submenu for the BSDs we’re covering today:

MENU TITLE PXE Boot Menu (Main)

LABEL bsd-oses
        MENU LABEL BSD Operating Systems
        KERNEL vesamenu.c32
        APPEND pxelinux.cfg/bsd

Now create the file referenced there:

# vi /usr/local/tftpboot/pxelinux.cfg/bsd

and put the following text in there:

MENU TITLE PXE Boot Menu (BSD)

LABEL main-menu
        MENU LABEL Main Menu
        KERNEL vesamenu.c32
        APPEND pxelinux.cfg/default

Alright, preparation work is done, let’s finally add some operating system data!

FreeBSD 12.2

PXE booting FreeBSD is actually not such an easy thing to do if you want to avoid using NFS shares. Fortunately there is mfsBSD, a project that provides tools as well as releases of special FreeBSD versions that can be booted over the net easily. We’re going to use that.

There are multiple variants for download: The standard one, the “special edition” and a “mini edition”. The special edition comes with the distribution tarballs on the ISO – you may want to use that one for installation purposes. If you just want a FreeBSD live system (e.g. for maintenance and repairs) or use you owr mirror (see below), the standard edition is for you since it is much smaller and thus boots way faster.

Let’s make the image available via HTTP:

# mkdir -p /usr/local/www/pxe/bsd/fbsd
# fetch https://mfsbsd.vx.sk/files/iso/12/amd64/mfsbsd-12.2-RELEASE-amd64.iso -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso
# gzip -9 /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso

Now edit the pxelinux.cfg/bsd file and append:

LABEL fbsd-pxe-install
        MENU LABEL Install FreeBSD 12.2 (PXE)
        MENU DEFAULT
        KERNEL memdisk
        INITRD http://10.11.12.1/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso
        APPEND iso raw

That’s it. You can now PXE-boot into FreeBSD.

Installation using mfsBSD

Login with user root and password mfsroot. It includes a “zfsinstall” script that you may want to take a look at. There’s a lot more to mfsBSD, though. Its tools allow you to easily roll your own customized images. If a way to include packages or files, use a custom-built kernel and things like that sounds like something that would be useful for you, take a closer look. I cannot go into more detail here – it’s a topic of its own and would deserve an entire article dedicated to it. In case you just want to use the familiar FreeBSD installer bsdinstall, read on.

Mirroring distfiles and fixing bsdinstall

If you want to install FreeBSD over PXE more than once, it makes sense to provide a local distfile mirror. Since we have a fileserver running anyway, there’s really nothing more to it than getting the distfiles and putting them into the right place. At the very minimum get the following three files:

# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/MANIFEST -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/MANIFEST
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/base.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/base.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/kernel.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/kernel.txz

Depending on which distfiles you usually install, also get any or all of the following files:

# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/base-dbg.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/base-dbg.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/kernel-dbg.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/kernel-dbg.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/lib32-dbg.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/lib32-dbg.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/ports.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/ports.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/src.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/src.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/tests.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/tests.txz

At this point in time, bsdinstall is broken on mfsBSD. The reason is that the distfile manifest is missing. I think about getting this fixed upstream, so in the future try and see if the following part is obsolete before using it. But for now, let’s create a simple shell script in the webroot directory for convenience:

# vi /usr/local/www/pxe/fbsd.sh

Put the following into the script:

#!/bin/sh
ARCH=`uname -m`
RELEASE=`uname -r | cut -d"-" -f1`
mkdir -p /usr/freebsd-dist
fetch http://10.11.12.1/bsd/fbsd/${ARCH}/${RELEASE}-RELEASE/MANIFEST -o /usr/freebsd-dist/MANIFEST
bsdinstall

Now if you PXE-booted mfsBSD and logged in as root, you just need to execute the following command line and will then be able to use the installer as you are used to it:

# fetch http://10.11.12.1/fbsd.sh && sh fbsd.sh

When you are to select the installation source, there is an “Other” button at the bottom of the window. Choose that and point to your distfile mirror – in my example http://10.11.12.1/bsd/fbsd/amd64/12.2-RELEASE. Happy installing!

One more hint: You may want to look into the environment variables that bsdinstall(8) provides. I briefly attempted to automatically set the URL to the distfile mirror but couldn’t get it working. As I was already running out of time with this article I haven’t looked deeper into it. If anybody figures it out I’d appreciate sharing your solution here.

OpenBSD 6.8

Adding OpenBSD as an option is trivial. The project provides a ramdisk kernel used for installing the system and a NBP capable of loading it. Let’s get those two files in place – and while the ramdisk kernel is fairly small already, we can chop a couple of bytes off by compressing it:

# mkdir -p /usr/local/tftpboot/bsd/obsd
# fetch https://cdn.openbsd.org/pub/OpenBSD/6.8/amd64/pxeboot -o /usr/local/tftpboot/bsd/obsd/pxeboot
# fetch https://cdn.openbsd.org/pub/OpenBSD/6.8/amd64/bsd.rd -o /usr/local/tftpboot/bsd/obsd/6.8-amd64.rd
# gzip -9 /usr/local/tftpboot/bsd/obsd/6.8-amd64.rd

Now we only need to add the required lines to pxelinux.cfg/bsd:

LABEL obsd-pxe-install
        MENU LABEL Install OpenBSD 6.8 (PXE)
        KERNEL pxechn.c32
        APPEND bsd/obsd/pxeboot

That’s it, the OpenBSD loader can be booted! Since we don’t have the kernel in the assumed default location (“/bsd”) we’d need to tell the loader to “boot bsd/obsd/6.8-amd64.rd.gz”. The loader supports a configuration file, though. So for a little extra convenience we can make it pick up the kernel automatically like this:

# mkdir -p /usr/local/tftpboot/etc
# ln -s /usr/local/tftpboot/etc/boot.conf /usr/local/tftpboot/bsd/obsd/boot.conf
# echo "boot bsd/obsd/6.8-amd64.rd.gz" > /usr/local/tftpboot/bsd/obsd/boot.conf
# echo "# OpenBSD boot configuration" >> /usr/local/tftpboot/bsd/obsd/boot.conf

The pxeboot program comes with the configuraton file name of etc/boot.conf hard-coded. To keep things a little cleaner in the hierarchy that I use, I chose to set a symlink in the obsd directory for reference purposes. And that’s all.

NetBSD 9.1

Let’s add NetBSD! It’s somewhat similar to OpenBSD – but a bit more involved unfortunately. The reason is that the NBP by default does not support a configuration file. It has the ability to use one, but that needs to be activated first. Which is fair enough since it’s only a single command – on NetBSD that is! But let’s worry about this in a minute and first get the NBP as well as the install kernel:

# mkdir -p /usr/local/tftpboot/bsd/nbsd
# fetch https://cdn.netbsd.org/pub/NetBSD/NetBSD-9.1/amd64/installation/misc/pxeboot_ia32.bin -o /usr/local/tftpboot/bsd/nbsd/pxeboot_ia32.bin
# fetch https://cdn.netbsd.org/pub/NetBSD/NetBSD-9.1/amd64/binary/kernel/netbsd-INSTALL.gz -o /usr/local/tftpboot/bsd/nbsd/netbsd-INSTALL.gz

Now we need to add the boot menu entry by adding the following lines to pxelinux.cfg/bsd:

LABEL nbsd-pxe-install
        MENU LABEL Install NetBSD 9.1 (PXE)
        KERNEL pxechn.c32
        APPEND bsd/nbsd/pxeboot_ia32.bin

This is sufficient to load and execute the NetBSD loader. That will then complain that it cannot find the kernel and no hints about NFS were given. Now we have three options:

1. Manually point the loader to the correct kernel each time
2. Give the required hint via DHCP
3. Try to enable the loader configuration

Typing in “tftp:bsd/nbsd/netbsd-INSTALL.gz” is probably fair enough if you are doing NetBSD installs very rarely but it gets old quickly. So let’s try out option two!

Modifying DHCP config for NetBSD

The DHCP server needs to be configured to pass a different Boot File name option when answering the NetBSD loader than otherwise. This is done by matching class information. This topic is beyond the scope of this article, so if you are interested, do some reading on your own. I won’t leave you hanging, though, if you just need to get things working.

Here’s what you have to add to the configuration if you’re using Kea – for example right before the “loggers” section:

    "client-classes": [
        {
            "name": "NetBSDloader",
            "test": "option[vendor-class-identifier].text == 'NetBSD:i386:libsa'",
            "boot-file-name": "tftp:bsd/nbsd/netbsd-INSTALL.gz"
        }
    ],

And here the same thing if you are using DHCPd:

if substring (option vendor-class-identifier, 0, 17) = "NetBSD:i386:libsa" {
    if filename = "netbsd" {
        filename "tftp:bsd/nbsd/netbsd-INSTALL.gz";
    }
}

Restart your DHCP server and you should be good to go.

After accomplishing the easy way via DHCP, I also went down to explore the boot.cfg road but ultimately failed. I’m documenting it here anyway in case somebody wants to pick up where I decided to leave it be.

Enabling boot.cfg in the loader

To mitigate the risk of polluting my main system by doing something stupid I chose to do all of this using my unprivileged user. The first thing I did, was fetching and extracting the basic NetBSD 9.1 sources:

% mkdir -p netbsd-9.1 && cd netbsd-9.1
% fetch ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-9.1/source/sets/src.tgz
% tar xvzf src.tgz

The sources for the installboot program we’re looking for are in usr/src/usr.sbin/installboot. I tried to get that thing to build by pointing the compiler at additional include directories and editing quite some header files, hoping to resolve the conflicts with FreeBSD’s system headers and problems like that. It can probably be done but that would take a C programmer – which I am not.

Fortunately NetBSD is the portability star among the BSDs and should be buildable on many other systems. I’ve never done this before but here was the chance. So I installed CVS and checked out the rest of the NetBSD src module:

% doas pkg install -y cvs
% cd netbsd-9.1/usr
% cvs -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -r netbsd-9-1-RELEASE -P src

When exploring the source tree, I found a build script that obviously does all the magic required here. Be warned however that this builds a full cross-toolchain including the complete GCC compiler! Then it builds the “tools” subset of the NetBSD code (which includes the installboot that we’re looking for). On my old and slow Atom-based system this process took 6 hours:

% cd src
% ./build.sh -U -m amd64 -T ~/nbsd tools
===> build.sh command:    ./build.sh -U -m amd64 -T /home/kraileth/nbsd tools
===> build.sh started:    Thu Feb  3 00:13:41 CET 2021
===> NetBSD version:      9.1
===> MACHINE:             amd64
===> MACHINE_ARCH:        amd64
===> Build platform:      FreeBSD 12.2-RELEASE-p1 amd64
===> HOST_SH:             /bin/sh
===> No $TOOLDIR/bin/nbmake, needs building.
===> Bootstrapping nbmake
checking for sh... /bin/sh
checking for gcc... cc

[...]

install ===> config
#   install  /home/kraileth/nbsd/bin/nbconfig
mkdir -p /home/kraileth/nbsd/bin
/home/kraileth/nbsd/bin/amd64--netbsdelf-install -c  -r -m 555 config /home/kraileth/nbsd/bin/nbconfig
===> Tools built to /home/kraileth/nbsd
===> build.sh ended:      Thu Feb  3 6:26:25 CET 2021
===> Summary of results:
         build.sh command:    ./build.sh -U -m amd64 -T /home/kraileth/nbsd tools
         build.sh started:    Thu Feb  4 07:13:41 CET 2021
         NetBSD version:      9.1
         MACHINE:             amd64
         MACHINE_ARCH:        amd64
         Build platform:      FreeBSD 12.2-RELEASE-p1 amd64
         HOST_SH:             /bin/sh
         No $TOOLDIR/bin/nbmake, needs building.
         Bootstrapping nbmake
         MAKECONF file:       /etc/mk.conf (File not found)
         TOOLDIR path:        /home/kraileth/nbsd
         DESTDIR path:        /usr/home/kraileth/netbsd-9.1/usr/src/obj/destdir.amd64
         RELEASEDIR path:     /usr/home/kraileth/netbsd-9.1/usr/src/obj/releasedir
         Created /home/kraileth/nbsd/bin/nbmake
         Updated makewrapper: /home/kraileth/nbsd/bin/nbmake-amd64
         Tools built to /home/kraileth/nbsd
         build.sh ended:      Thu Feb  4 12:26:25 CET 2021
===> .

The “-U” flag enables some trickery to build as an unprivileged user. With “-m” you specify the target architecture (I did use i386 but modified the above lines as that will be what most people will want to use instead!). Finally the “-T” switch allows to specify the installation target directory and the “tools” is the make target to use.

When it was done, I did the following (as root):

# cp /usr/local/tftpboot/bsd/nbsd/pxeboot_ia32.bin /usr/local/tftpboot/bsd/nbsd/pxeboot_ia32.bin.bak
# /usr/home/kraileth/netbsd-9.1/usr/src/tools/installboot/obj/installboot -eo bootconf /usr/local/tftpboot/bsd/nbsd/pxeboot_ia32.bin

This should enable the boot config file on the pxeboot loader. It does change the file and probably even makes the right change. I tried to enable module support via installboot, too and that obviously worked (the NFS module was loaded the next time I chainloaded the NetBSD loader). But for some reason I could not get boot.cfg to do what I wanted. Probably I don’t understand the file properly…

While it’s a bit disappointing to stop so close to the goal, messing with NetBSD so much already took much more time away from the other BSDs than I had imagined. And since I could at least offer a working way this was when I decided to move on.

DragonFly BSD

I attempted to get DragonFly BSD to work but failed. I briefly tried out a setup that includes NFS shares but it didn’t work completely either: Kernel booted but failed to execute /sbin/init for some reason or another. Also I don’t really want to cover NFS in this series – there’s enough material in here already. And without NFS… Well, DragonFly BSD has the same problem that FreeBSD has: It will boot the kernel but then be unable to mount the root filesystem.

While I guess that the mfsBSD approach could likely work for DF, too, this is something much more involved than reasonable for our topic here. I would really like to cover DragonFly here, too, but that’s simply a bit too much. If anybody knows how to get it working – please share your knowledge!

HardenedBSD 12-STABLE

HardenedBSD being a FreeBSD fork, inherited the same characteristics as vanilla FreeBSD. Which means that PXE booting the standard images is not an easy thing to do. HardenedBSD uses the same installer, bsdinstall however and for that reason it’s possible to install HardenedBSD by using mfsBSD as prepared above in the FreeBSD section. We only need to point the installer to a different distfile mirror. Let’s create that one now:

# mkdir -p /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE
# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/MANIFEST -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/MANIFEST
# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/base.txz -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/base.txz
# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/kernel.txz -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/kernel.txz

As with FreeBSD, there are some optional distfiles you may or may not want to mirror, too. Provide what you need for your installations:

# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/base-dbg.txz -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/base-dbg.txz
# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/kernel-dbg.txz -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/kernel-dbg.txz
# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/src.txz -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/src.txz
# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/tests.txz -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/tests.txz

Now we’re creating a convenience script for HardenedBSD:

# vi /usr/local/www/pxe/hbsd.sh

Put the following into the script:

#!/bin/sh
ARCH=`uname -m`
MAJOR=`uname -r | cut -d"." -f1`
mkdir -p /usr/freebsd-dist
fetch http://10.11.12.1/bsd/hbsd/${ARCH}/${MAJOR}-STABLE/MANIFEST -o /usr/freebsd-dist/MANIFEST
bsdinstall

Now fire up mfsBSD, login as root and simply run the following command line to start the installer:

# fetch http://10.11.12.1/hbsd.sh && sh hbsd.sh

Select the “Other” button when asked for the installation source. Choose that and point to your distfile mirror – in my example http://10.11.12.1/bsd/hbsd/amd64/12-STABLE. And that’s it.

MidnightBSD 2.0

Since MidnightBSD is a FreeBSD fork as well, it also suffers from the well-known problems related to PXE booting. Again mfsBSD comes to the rescue. Let’s create the distfile mirror first:

# mkdir -p /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/MANIFEST -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/MANIFEST
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/base.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/base.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/kernel.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/kernel.txz

Pick any or all of the remaining optional distfiles to be mirrored, too, if you need them:

# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/lib32.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/lib32.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/doc.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/doc.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/base-dbg.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/base-dbg.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/kernel-dbg.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/kernel-dbg.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/lib32-dbg.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/lib32-dbg.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/src.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/src.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/mports.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/mports.txz

And here’s the convenience script:

# vi /usr/local/www/pxe/mbsd.sh

Put the following into it:

#!/bin/sh
ARCH=`uname -m`
RELEASE="2.0"
mkdir -p /usr/freebsd-dist
fetch http://10.11.12.1/bsd/mbsd/${ARCH}/${RELEASE}-RELEASE/MANIFEST -o /usr/freebsd-dist/MANIFEST
bsdinstall

Now you can PXE-boot mfsBSD as prepared in the FreeBSD section. After logging in as root execute the following command line that will take you to the installer:

# fetch http://10.11.12.1/mbsd.sh && sh mbsd.sh

When given the choice to select the installation source make sure to select “Other” and point to the right URL – in my example it would be http://10.11.12.1/bsd/mbsd/amd64/2.0-RELEASE, then install the OS as you’re used to.

Watch out for some options in the installer, though! MidnightBSD is mostly on par with FreeBSD 11.4. If you enable e.g. newer hardening options that the installer knows but the target OS doesn’t, you might run into trouble (not sure, tough, I didn’t think of this until after my test installation).

What’s next?

I didn’t plan this post to get that long, especially NetBSD surprised me. Eventually I decided to cover HardenedBSD and MidnightBSD as well and accept that this is a BSD-only article. So the next one will add various Linuxen and other operating systems to the mix.

Multi-OS PXE-booting from FreeBSD 12: Required services (pt. 2)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/multi-os_pxe-booting_from_fbsd_pt2.gmi

The previous post was about what lead me to do this in the first place, featured a little excursion for people new to PXE and most importantly detailed the setup of a FreeBSD router that will be turned into a PXE server in this article.

While I originally intended to show how to boot Ubuntu in this part, things have changed a little. I realized that the software choices I made might not be what a lot of people would have chosen. Therefore I did a little extra work and present my readers with multiple options. This made the article long enough even without the Ubuntu bits which I wanted to cover in part part 3 instead (it was moved to part 4, however).

Current state and scope

Today we will make the machine offer all the services needed for network installations of many Open Source operating systems. My examples are all IPv4, but it should be possible to adapt this to IPv6 fairly easily. As this is *nix, there’s always multiple ways to do things. For software that does not come included in FreeBSD’s base installation, I’ll be giving two options. One will be less commonly deployed but have some advantages in my book. The other will achieve the same goal with a more popular solution.

The machine that I use is an old piece of metal that has 6 NICs – which is why I like to still use it for tinkering with network-related stuff. After part 1, we left off with a gateway that has an Internet-facing adapter em0 which gets its IP via DHCP from my actual router. We’re also using em5 which is statically configured to have the IP 10.11.12.1 and is connected to a separate switch. There’s an unbound nameserver running and serving that interface and a pf firewall is active doing NAT.

This means that everything is in place to serve any host connected to said switch, if it’s manually configured to use a static address in the 10.11.12.0/24 IP range and with default router and nameserver set to 10.11.12.1. Let’s start by getting rid of that “manually configured” requirement!

Excursion: DHCP basics

We do so by configuring the machine as a DHCP server handing out IP addresses for the 10.11.12.0/24 subnet on the 6th NIC. DHCP servers work by listening for DHCP requests which are broadcasted on the network (as the client does not have it’s own IP, yet). When receiving one, it will have a look at its configuration: Is there anything special for the host asking? Is the MAC address included with the request maybe blacklisted? Is there a reserved IP to be handed to this specific machine? Or any particular option to send to the “class” of device asking?

In our simple case there aren’t really any bells and whistles involved. So it will have a look at the IP pool that it manages and if it can find an unused one, it will answer with a DHCP offer. The latter is a proposal for an IP to be leased by the client and also includes various network-related information: Usually at least the netmask, router and nameserver. A lot of additional information can be provided in form of options; you can point the client joining the network to a time server if you have one, inform about the domain being used and much, much more (even custom options are possible if you need them).

For PXE booting to work we need to make use of two particular options: We need the PXE code in the firmware to know which server to turn to if it wants to load the NBP (Network Bootstrap Program) from the net. It also needs to know what the file to ask for is called.

DHCP servers

There are multiple DHCP servers out there. If I were doing this on Linux, I’d probably just pick Dnsmasq and be done: As the name implies, it does DNS. But it also does DHCP and TFTP (which we are in need of as well) and supports PXE. But FreeBSD comes with its own TFTP daemon that I’m going to use and I actually prefer the Unix way over all-rounder software: Do one thing and do it well!

The first thing that comes to mind in terms of DHCP servers is ISC’s DHCPd. It’s small, simple to use (at least for our use case), battle-tested and in use pretty much everywhere. It’s old, though, not extremely fast and certainly not as flexible as you might wish for today. This (among other things) lead the ISC to start a new project meant as a replacement: Kea.

The latter is a modern DHCP server with a lot of nice new features: It is a high-performance solution that’s extensible, supports databases as backends, has a web GUI (Stork) available and more. But since DHCPd works well enough, adoption of Kea has been slow. There are a couple of downsides to it, too: First and foremost – its configuration is written in JSON. Yes, JSON! While there are legitimate use cases for that format, configuration is not one of them if you ask me. That was a terrible choice. Kea also pulls in big dependencies like the boost C++ libraries not everybody is fond of.

IMO the benefits of Kea outweight the drawbacks (if it wasn’t for the JSON configuration, I’d even state: clearly). But it’s your choice of course.

DHCP server option 1: Modern-day Kea

Alright, let’s give Kea a try, shall we? First we need to install it and then edit the configuration file:

# pkg install -y kea
# vi /usr/local/etc/kea/kea-dhcp4.conf

The easiest thing to do is to delete the contents and paste the following. Then adapt it to your network and save:

{
"Dhcp4": {
    "interfaces-config": {
        "interfaces": [ "em5/10.11.12.1" ]
    },
    "control-socket": {
        "socket-type": "unix",
        "socket-name": "/tmp/kea4-ctrl-socket"
    },
    "lease-database": {
        "type": "memfile",
        "lfc-interval": 3600
    },
    "expired-leases-processing": {
        "reclaim-timer-wait-time": 10,
        "flush-reclaimed-timer-wait-time": 25,
        "hold-reclaimed-time": 3600,
        "max-reclaim-leases": 100,
        "max-reclaim-time": 250,
        "unwarned-reclaim-cycles": 5
    },

    "renew-timer": 900,
    "rebind-timer": 1800,
    "valid-lifetime": 3600,
    "next-server": "10.11.12.1",
    "boot-file-name": "pxelinux.0",

    "option-data": [
        {
            "name": "subnet-mask",
            "data": "255.255.255.0"
        },
        {
            "name": "domain-name-servers",
            "data": "10.11.12.1"
        },
        {
            "name": "domain-name",
            "data": "example.org"
        },
        {
            "name": "domain-search",
            "data": "example.org"
        }
    ],

    "subnet4": [
        {
            "subnet": "10.11.12.0/24",
            "pools": [ { "pool": "10.11.12.20 - 10.11.12.30" } ],
            "option-data": [
                {
                    "name": "routers",
                    "data": "10.11.12.1"
                }
            ]
        }
    ],

    "loggers": [
    {
        "name": "kea-dhcp4",
        "output_options": [
            {
                "output": "/var/log/kea-dhcp4.log"

            }
        ],
        "severity": "INFO",
        "debuglevel": 0
    }
  ]
}
}

Yes, looks pretty bad, I know. But that’s only the representation; if something better had been used (say YAML), it’d be about 50 lines instead of 75, be much more readable and above all: less error-prone to edit. Oh well. If you can ignore the terrible representation, the actual data is not so bad and pretty much self-explaining.

I’d like to point you at the “next-server” and “boot-file-name” global options that I set here. These are required for PXE booting by pointing to the server hosting the NBP and telling its file name. Leave them out and you will still have a working DHCP server if you don’t need to do PXE. While this configuration works, you will likely want to extend it for production use.

With the config in place, let’s enable and start the daemon:

# sysrc kea_enable="YES"
# service kea start

A quick look if a daemon successfully bound to port 67 and is listening doesn’t hurt:

# sockstat -4l | grep 67
root     kea-dhcp4  1480  14 udp4   10.11.12.1:67         *:*

Ok, there we are. We now have a DHCP service on our internal network!

DHCP server option 2: Venerable ISC DHCPd

So you’d like to play simple and safe? No problem, going with DHCPd is not a bad choice. But first we need to install it and edit the configuration file:

# pkg install -y isc-dhcp44-server
# vi /usr/local/etc/dhcpd.conf

Delete everything. Then add the following (adjust to your network structure, of course!) and save:

option domain-name "example.org";
option domain-name-servers 10.11.12.1;
option subnet-mask 255.255.255.0;
 
default-lease-time 600;
max-lease-time 7200;
ddns-update-style none;
log-facility local7;

next-server 10.11.12.1;
filename "pxelinux.0";
 
subnet 10.11.12.0 netmask 255.255.255.0 {
  range 10.11.12.10 10.11.12.20;
  option routers 10.11.12.1;
}

Mind the “next-server” and “filename” options which define the server to get the NBP from as well as the file name of that. You can leave out that block and will still have a working DHCP server – but it won’t allow for PXE booting in that case. I’d also advice you to do a bit of reading and probably do a more comprehensive configuration of DHCPd.

Next thing to do is to enable DHCPd, confine it to serve requests coming in from one particular NIC only and start the service:

# sysrc dhcpd_enable="YES"
# sysrc dhcpd_ifaces="em5"
# service isc-dhcpd start

Quick check to see if the service is running and binding on port 67:

# sockstat -4l | grep 67
dhcpd    dhcpd      1396  8  udp4   *:67                  *:*

Looking good so far, DHCP should be available on our internal net.

Optional: Checking DHCP

If you want to make sure that your DHCP server is not only running but that it can also be reached and actually does what it’s meant to, you can either just try to power up a host in the 10.11.12.0/24 network and configure it to get its IP via DHCP. Or you can for example use the versatile nmap tool to test DHCP discovery from any host on that network:

# pkg install -y nmap
# nmap --script broadcast-dhcp-discover
Starting Nmap 7.91 ( https://nmap.org ) at 2021-01-24 17:56 CET
Pre-scan script results:
| broadcast-dhcp-discover: 
|   Response 1 of 1: 
|     Interface: em0
|     IP Offered: 10.11.12.21
|     DHCP Message Type: DHCPOFFER
|     Subnet Mask: 255.255.255.0
|     Router: 10.11.12.1
|     Domain Name Server: 10.11.12.1
|     Domain Name: example.org
|     IP Address Lease Time: 1h00m00s
|     Server Identifier: 10.11.12.1
|     Renewal Time Value: 15m00s
|_    Rebinding Time Value: 30m00s
WARNING: No targets were specified, so 0 hosts scanned.
Nmap done: 0 IP addresses (0 hosts up) scanned in 10.56 seconds

# pkg delete -y nmap

All right! DHCP server is working and happily handing out leases.

TFTP

The Trivial File Transfer Protocol daemon is up next. FreeBSD ships with a TFTP daemon in the base system, so we’re going to use that. It will not be used by itself but instead from the inetd super daemon. To enable TFTP, we just need to put one line in the inetd configuration file:

# echo "tftp    dgram   udp     wait    root    /usr/libexec/tftpd      tftpd -l -s /usr/local/tftpboot" >> /etc/inetd.conf

Now we need to create the directory that we just referenced, as well as a subdirectory which we’re going to use and create a file there:

# mkdir -p /usr/local/tftpboot/pxelinux.cfg
# vi /usr/local/tftpboot/pxelinux.cfg/default

Put the following in there (for now) and save:

DEFAULT vesamenu.c32
PROMPT 0

All is set, let’s enable and start the service now:

# sysrc inetd_enable="YES"
# service inetd start

Again we can check real quick if the service is running:

# sockstat -4l | grep 69
root     inetd      1709  6  udp4   *:69                  *:*

Good, so now we can test to fetch the configuration file from either our server or from any FreeBSD machine in the 10.11.12.0/24 network:

# tftp 10.11.12.1
tftp> get pxelinux.cfg/default
Received 292 bytes during 0.0 seconds in 1 blocks
tftp> quit
# rm default

Everything’s fine just as expected.

File Server

The remaining piece we need to set up is a means to efficiently transfer larger files over the wire – i.e. not TFTP! You can do it via FTP and use FreeBSD’s built-in FTP daemon. While this works well, it is not the option that I’d recommend. Why? Because FTP is an old protocol that does not play nicely with firewalls. Sure, it’s possible to do FTP properly, but that’s more complex to do than using something else like a webserver that speaks HTTP.

If you want to go down that path, there are a lot of options. There’s the very popular and feature-rich Apache HTTPd and various more light-weight solutions like LighTTPd and many more. I generally prefer OpenBSD’s HTTPd because it is so easy to work with – and when it comes to security or resisting feature creep its developers really mean it. If I need to do something that it cannot do, I usually resort to the way more advanced (and much more popular) Nginx.

Pick any of the two described here, go with FTPd instead or just ignore the following three sections and set up the webserver that you prefer to deploy.

If you didn’t opt for FTP, as a first step create a directory for the webserver and change the ownership:

# mkdir -p /usr/local/www/pxe
# chown -R www:www /usr/local/www/pxe

File Server option 1: OpenBSD’s HTTPd

Next is installing the program and providing the desired configuration. Edit the file:

# pkg install -y obhttpd
# vi /usr/local/etc/obhttpd.conf

Delete the contents and replace it with the following, then save:

chroot "/usr/local/www"
logdir "/var/log/obhttpd"
 
server "pixie.example.org" {
        listen on 10.11.12.1 port 80
        root "/pxe"
        log style combined
}

Super simple, eh? That’s part of the beauty of obhttpd. OpenBSD follows the “sane defaults” paradigm. That way you only have to configure stuff that is specific to your task as well as things where you want to change the defaults. Surprisingly, this configuration does it – and there’s really not much I’d change for a production setup if it is the only site on this server.

It’s always a good idea to check if the configuration is valid, so let’s do that:

# obhttpd -nf /usr/local/etc/obhttpd.conf
configuration OK

If you ever need to debug something, you can start the daemon in foreground and more verbosely by running obhttpd -dvv. Right now the server would not start because the configured log directory does not exist. So this would be a chance to give debugging a try.

Let’s create the missing directory and then enable and start the service:

# mkdir /var/log/obhttpd
# sysrc obhttpd_enable="YES"
# service obhttpd start

As always I prefer to take a quick look if the daemon did bind the way I wanted it to:

# sockstat -4l | grep httpd
www      obhttpd    1933  7  tcp4   10.11.12.1:80         *:*
www      obhttpd    1932  7  tcp4   10.11.12.1:80         *:*
www      obhttpd    1931  7  tcp4   10.11.12.1:80         *:*

Looks good.

File Server option 2: Nginx

Next thing is installing Nginx and providing a working configuration:

# pkg install -y nginx
# vi /usr/local/etc/nginx/nginx.conf

Erase the example content and paste in the following:

user  www;
error_log  /var/log/nginx/error.log;
worker_processes  auto;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    keepalive_timeout  65;

    server {
        listen       80;
        location / {
            root   /usr/local/www/pxe;
        }
    }
}

This is by no means a perfect configuration but only an example. If you want to deploy Nginx in production, you’ll have to further tune it towards what you want to achieve. But now let’s enable and start the daemon:

# sysrc nginx_enable="YES"
# service nginx start

Time for the usual quick check:

# sockstat -4l | grep nginx
www      nginx      1733  6  tcp4   *:80                  *:*
www      nginx      1732  6  tcp4   *:80                  *:*
root     nginx      1731  6  tcp4   *:80                  *:*

Nginx is running and listening on port 80 as it should be.

File Server option 3: FTPd

FTP for you, eh? Here we go. FreeBSD comes with an ftp group but not such a user by default. Let’s create it:

# pw useradd -n ftp -u 14 -c "FTP user" -d /var/ftp -g ftp -s /usr/sbin/nologin

It’s a convention that public data offered via anonymous FTP is placed in a “pub” directory. We’re going to honor that tradition and create the directory now:

# mkdir -p /var/ftp/pub
# chown ftp:ftp /var/ftp/pub

If you intend to use logging, create an empty initial log file:

# touch /var/log/ftpd

Now we need to enable the FTP service for inetd (the “l” flag enables a transfer log, “r” is for operation in read-only mode, “A” allows for anonymous access and “S” is for enabling the download log):

# echo "ftp     stream  tcp     nowait  root    /usr/libexec/ftpd       ftpd -l -r -A -S" >> /etc/inetd.conf

As we intend to run a service that allows local users to log in via FTP, we need to consider the security implications of this. In my case I have created the “kraileth” user and it has the power to become root via doas. While OpenSSH is configured to only accept key-based logins, FTP is not. The user also has a password set – which means that an attacker who suspects that the user might exist, can try brute-forcing my password.

If you’re the type of person who is re-using passwords for multiple things, take this scenario into consideration. Sure, this is an internal server and all. But I recommend to get into a “security first” mindset and just block the user from FTP access, anyway. To do so, we just need to add it to the /etc/ftpusers file:

# echo "kraileth" >> /etc/ftpusers

Now let’s restart the inetd service (as it’s already running for TFTP) and check it:

# service inetd restart
# sockstat -4l | grep 21
root     inetd      1464  7  tcp4   *:21                  *:*

Ready and serving!

Optional: Checking File Server service

Time for a final test. If you’re using a webserver, do this:

# echo "TestTest" > /usr/local/www/pxe/test
# fetch http://10.11.12.1/test
test                                             9  B 8450  Bps    00s
# cat test
TestTest
# rm test /usr/local/www/pxe/test

If you’re using FTP instead:

# echo "TestTest" > /var/ftp/pub/test
# fetch ftp://10.11.12.1/pub/test
test                                                     9  B 9792  Bps    00s
# cat test
TestTest
# rm test /var/ftp/pub/test

Everything’s fine here, so we can move on.

What’s next?

In part 3, we’re finally going to add data and configuration to boot multiple operating systems via PXE.

Multi-OS PXE-booting from FreeBSD 12: Introduction (pt. 1)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/multi-os_pxe-booting_from_fbsd_pt1.gmi

This is an introductory article; if you’re familiar with PXE you will want to skip the excursion but may be interested in the “Why”. The article ends with the post-installation setup of my test machine, turning it into a simple router so that the next article can start with the actual PXE-related setup.

Also see part 2 here and part 3 here as well as part 4 here.

Situation

These days I decided to reactivate an old laptop of mine. It has processor of an older generation (Ivy Bridge) but it’s an i7 – and it has 24 GB of RAM, making it a somewhat nice machine for certain things. Problem with it is: The USB does not work (I think that the controller was fried)!

So how do I get a current OS installed on there as I cannot use the CD emulator I’d normally use? Sure, I could always burn a CD, but I don’t feel like it (unnecessary waste). I could open it up, remove the drive, place it in another machine, install the OS and then put it back. Nah, not too much fun, either. How about doing a network install then?

When I thought of that I realized that I had some old notes from about a year ago somewhere on my workstation. I originally meant to blog about that topic but never actually wrote the post. So here’s how to do what the post’s title says – updated to FreeBSD 12.2 and Ubuntu 20.04!

While I have a clear favorite OS and very much appreciate what FreeBSD has to offer for me as the administrator of a fleet of servers, there really is no reason to turn a blind eye to other Unix-like systems. For one thing it totally makes sense to always pick the best candidate for a job (which might not in all cases be one OS). That and the fact that you cannot judge which one is best suited if you don’t have at least some level of familiarity with various options. But in addition to that my employer runs a heterogeneous environment, anyway – and while I’m mostly occupied with the BSD side of things, there’s simply way, way too many Linux machines around to even think about avoiding them altogether all the time.

Why PXE-boot Linux from FreeBSD?

After an update, the old means of installing Linux servers as we do it at work had stopped working reliably. I looked at it briefly but soon found that too many things weren’t set up like you’d do it today. Therefore I decided that it might make more sense to start fresh. And while at it – wouldn’t it make sense to try and combine the Linux and FreeBSD PXE servers on one machine? That would mean one less server to run after all.

The next installation due was for a customer who requested an Ubuntu server. As a matter of fact we are transitioning to use that distribution more often for our internal infrastructure, too (decision by management, certainly not my choice!). For that reason one weekend I ended up doing something that I hadn’t done in a while: installing a fresh Ubuntu 16.04 system on a test machine I. After doing this I could write a whole post about how bad the installer actually is (static IP addressing, anyone??), but I don’t want this to turn into a rant.

So let’s just mention one single complaint: Installing takes quite a long time (I never understood why Debian and derivatives install stuff that they remove again later during the “cleaning up” phase…). Before the installation was even finished, I admittedly already had mixed feelings about this new system. But then this was what happened on the first boot:

[ TIME ] Timed out waiting for device dev-disk-by\x2duuid-0ef05387\x2d50d9\x2d4cac\x2db96\x2d9808331328af.device.
[DEPEND] Dependency failed for /dev/disk/by-uuid/0ef05387-50d9-4cac-b796-9808331328af.
[DEPEND] Dependency failed for Swap.
[  *** ] A start job is running for udev Coldplug all Devices (3min 34s / no limit)

You’ve got to be kidding! A freshly installed system going sideways like that? Sorry Ubuntu that’s not the kind of stuff that I like wasting my time with! I don’t even care if it’s systemd’s fault or something else. The boss “preferably” wanted an Ubuntu system – I gave it a try and it failed me. Ah, screw it, let’s deploy something I know of that it actually works (same hardware BTW, before anybody wants to blame that for a problem that’s definitely Ubuntu’s fault)!

I had a freshly installed FreeBSD with static IP configuration (which you probably want to use for a DHCP / PXE boot server after all) in a fraction of the time that the Ubuntu installation took. And it goes without saying: System starts as one would expect.

Excursion: PXE – An Overview

While there have been earlier methods for making a machine boot over the network, PXE (Preboot eXecution Environment) is what you want to rely on if you need to do that today. It is a specification widely implemented (in fact just about everywhere) and chances are very low that you will run into problems with it. Have a look around in your computer’s EFI or BIOS, often PXE-booting is disabled (and if you want to use it just once to install an OS on the machine, I recommend that you disable it again afterwards). How to make a machine boot over the net depends on its EFI / BIOS. Figure out how to do it four your metal and give it a try.

On your average home environment not too much should happen. The PXE environment will probe the network for the information that it needs, receive none and eventually give up. But what information does it need and which components are involved?

Well, it needs to load an operating system “from the net” – which means from another machine. To be able to talk to other hosts on the network, it needs to become part of said net. For this it needs to know the network parameters and requires a usable, unique IP address for itself. It can get all of that from a DHCP (Dynamic Host Configuration Protocol) server. If configured correctly, the daemon will accept DHCP requests and provide the client with a suggested IP as well as information about the network (like the netmask, default router, nameservers on the net, etc). It can also tell the client from where it can load the NBP (Network Bootstrap Program).

The latter is a piece of software, usually a special bootloader. It will be downloaded via TFTP (Trivial File Transfer Protocol). Think of it as very similar to FTP but much, much more primitive. The NBP is executed when the transfer is completed. It will then download its configuration file and then act accordingly, most likely fetching an operating system kernel and additional files, then booting the kernel.

TFTP is what the PXE bootcode speaks and uses. But due to its very simplistic nature, TFTP is not well fit for larger file transfers. Therefore such files are usually loaded via other means once the kernel has booted and more options are available. FreeBSD likes to use NFS, while on Linux HTTP is often used to receive a filesystem or installation medium image.

So the following services are involved when setting up a PXE boot server:

• DHCP server
• TFTP server
• Webserver / NFS service stack

Preparing a FreeBSD router

Today I’m redoing my previous work in my home lab with FreeBSD 12.2. The machine that I’m using is pretty old (32-bit only Atom CPU!) but it has 6 Ethernet ports and still works well for network tinkering. I got it for free some years ago, so there’s really no reason to complain.

When it comes to the installation, in this case I’m going with MBR + UFS which I normally wouldn’t do for a amd64 machine. It’s a standard 12.2 installation otherwise with my “kraileth” user (member of the wheel group) added during the installation.

First thing to do is copying my public SSH key onto the server and then SSHing into the machine:

% ssh-copy-id -i ~/.ssh/id_ed25519 kraileth@hel.local
% ssh kraileth@hel.local

Now I switch to the root user, disallow SSH password-based login and restart the SSH daemon:

% su -l
# sed -i "" 's/#ChallengeResponseAuthentication yes/ChallengeResponseAuthentication no/' /etc/ssh/sshd_config
# service sshd restart

I don’t need sendmail on the machine, so off it goes:

# sysrc sendmail_enable="NONE"

Next is bootstrapping the package manager, installing the unbound DNS server and – for convenience – doas (a simpler sudo replacement). Writing a one-line config for doas is fine for my use case:

# env ASSUME_ALWAYS_YES=TRUE pkg bootstrap
# pkg install -y doas unbound
# echo "permit :wheel" > /usr/local/etc/doas.conf

Then unbound needs to be configured. To do so, edit the config file like this:

# vi /usr/local/etc/unbound/unbound.conf

Delete all the content, put the following in there (adjust to your network, of course) and save:

server:
        verbosity: 1
        interface: 10.11.12.1
        access-control: 10.11.12.0/24 allow

Keep in mind that this is not a production configuration! You will have to do some research on the program if you want to put it into production. Do at the very least read the more than thousand lines of comments that you just deleted (you can find it in /usr/local/etc/unbound/unbound.conf.sample). The above config will make the daemon bind on the IP configured via the “interface” statement and allow DNS recursion for any host in the subnet defined via “access-control”. Let’s enable the daemon now (so it will start after rebooting the machine):

# sysrc unbound_enable="YES"

During installation I chose the first NIC (em0) to be configured via DHCP as offered by my home router. This machine will act as a DHCP server for another network, so I assign a static address to the respective interface (em5) that I have connected to a spare switch I had around. It will also act as a gateway between the two networks it is part of:

# sysrc ifconfig_em5="inet 10.11.12.1 netmask 255.255.255.0"
# sysrc gateway_enable="YES"

To actually provide Internet connectivity to hosts in the 10.11.12.0/24 subnet, we have to enable NAT (Network Address Translation). Otherwise any host that is contacted via an IP in that range will have no idea how to answer, even though our gateway forwarded the message to it. NATing is done via the firewall. FreeBSD offers three of them. Pf is my absolute favorite, but ipfw is ok, too. I wouldn’t recommend ipf – it’s old and there is really no reason to use it if you’re not already doing so and don’t want to switch.

First we’ll create a configuration file:

# vi /etc/pf.conf

Now put the following in there (again: Adjust to your network) and save:

ext_if="em0"
int_if="em5"
localnet = $int_if:network

scrub in all
nat on $ext_if from $localnet to any -> ($ext_if)

This defines commonly used three macros: One for the external and internal NIC and one for the local network. It then turns scrubbing on (you almost always want to do this and I suggest that you look up what it does). The final line does the magic of the actual NAT.

You will want to define a more complex firewall configuration if you plan to use this gateway in production. But this is enough for my (demonstration) purposes. I suggest that you follow along and when you made sure that PXE booting works, tighten up security – then test if everything still works. Firewalling is a topic of its own, and in fact not a small one, though. If you’re new to it it makes perfect sense to do some reading. There’s lots of free material on the net as well as a great “Book of Pf” by Peter Hansteen if you prefer.

So let’s enable Pf on startup:

# sysrc pf_enable="YES"
# sysrc pf_rules="/etc/pf.conf"

Reminder: If you want to do things right, you will probably want to enable pflog, too, for example. This post is meant to point you in the right direction and to get stuff to work quickly. Familiarizing yourself with the various components used is your own homework.

It’s always a good idea to update the system to the latest patchlevel:

# freebsd-update fetch install
# freebsd-version -kr
12.2-RELEASE-p1
12.2-RELEASE

Ok, looks like kernel-related updates were installed, so it’s time to reboot:

# shutdown -r now

That’s it for the preparation work. Once the server comes back up again it should function as a router. If you configure a host with a static address in the 10.11.12.0/24 network, it should be able to have DNS queries answered as well as reach the Internet.

What’s next?

In part 2 we’re going to complete the setup of the PXE boot server so that a client can boot into Ubuntu Linux. The rest will be in part 3.