How to choose your *BSD OS to begin with?

After publishing my previous post about how FreeBSD’s documentation is way better than that of Linux, I was asked which BSD I’d recommend to somebody who knows Linux but is new to the world of *BSD. This is a good question – and definitely one that cannot be answered in just a comment reply. So here’s a whole article about that topic (and in fact a rather lengthy one).

The question has been asked many times on the net and a lot of people have given different answers. Many of those recommendations are rather old now and the points that they made may no longer apply. Also it’s often BSD veterans who – of course – tend to advocate their BSD of choice. I’m a newcomer myself and so far I have not settled on one particular BSD but still try to explore all of them, their strong points and their shortcomings. If you ask me about Linux, I’m tempted to recommend the distro that I’ve been using for years now. When it comes to the BSDs this is not the case, yet, and while I do not make the claim that I’m perfectly neutral, I think that I can write something that could be of some value to people thinking about giving some BSD a try.

Dear Linux user!

Long-time Linux users will probably remember the famous “Welcome letter to a Windows user” from 2006. It’s a real classic which is outdated in some regards but the main points are still very much valid. The most important one (which applies in our case as well): If you’re interested in another operating system, it is imperative to be open-minded and to give this OS’s way of doing things a chance! It’s not a good idea at all to judge from the perspective of your old OS. Quite some things will be different. And as a matter of fact, you won’t be able to judge if certain decisions were sensible – they may seem totally weird to you but then actually make perfect sense if you know the background (which you don’t until you gain much more knowledge about the design of the system).

All BSDs are direct descendents of the original AT&T UNIX. Compared to Linux they have a much longer history and there are lots and lots of things that can probably only be explained if you go back in time far enough. While all BSDs share a common heritage they have of course been modernized over time. That is why they also have diverged quite a bit over the last two decades (and why it makes sense to write an article like this!).

One of the most important differences between the Linux world and the BSD one is that there are no “BSD distributions”. Sure, you will find some people use that term on the net but it’s most likely wrong. Most likely? Yes, because there arguably are BSD distributions… But that’s a very special case. You definitely know Debian and you probably know Gentoo. These are two Linux distros, right? Sure – but they could also be called “BSD distributions”. Why? Because there’s not only Debian GNU/Linux. The Debian project also offers Debian GNU/Hurd. And there’s Debian GNU/kFreeBSD, too! The latter is a typical Debian distribution (GNU userland) running on top of the FreeBSD kernel. For Gentoo it’s the same story. That’s where there’s something that you may call a “BSD distribution”. But that’s a rather Linuxy point of view!

While Linux is just a kernel and it is the distributions that create a full operating system by combining it with a userland (usually GNU), all BSDs follow a “whole system“ approach. There’s no such thing as “the BSD kernel”. Each BSD has its own. Fedora and Arch Linux may use different versions and configure their kernels differently, but it’s the same Linux kernel from the same sources that they use. The FreeBSD kernel and the OpenBSD kernel for example are entirely different kernels (even though they share common ancestry). It’s due to this “whole system” approach that calling e.g. NetBSD a “distribution” is plain wrong (and for the same reason BSD people will probably call the idea of a “BSD distribution” an absurdity in the first place).

What else should you know? Well, there’s a lot of little things that will be different from what you are used to. Drives and network interfaces have different names (like ada0p1 instead of sda1, em0 instead of eth0 or enp0s3). Some commands have different options and/or switches. Filesystems are different (unless you’re going to try FreeBSD and already used ZFSonLinux). The configuration works differently and you’re going to meet a different init system. You’ll probably come across a lot of things that turn out not to work like you intended. But that’s nothing horrible. You’re familiar with one unix-like system (Linux) and the BSDs are all unix-likes themselves after all!

A potential problem is drivers. If you always use the absolutely newest hardware (or pretty exotic stuff), chances are that Linux may fit you better. All of the BSD projects are much smaller than Linux is and so it is impossible for them to support as much hardware. However they are doing a fantastic job for the manpower that they have. Just try it out and see!

FreeBSD

Why should you try FreeBSD? FreeBSD is the BSD operating system with the largest community. This can be an important point for deciding which BSD to give a try. Thanks to its relatively large community it offers by far the widest range of software available. FreeBSD is the base of a lot of commercial products and appliances and for that reason it receives quite a bit of funding from the industry. That money can then be spent to fund projects that nobody volunteers for.

FreeBSD hat good support for technology like EFI. Nvidia officially supports FreeBSD and provides a closed-source driver that is meant to be on-par with the Windows driver (this can be interesting if you need good graphics performance). If you are into virtualization, FreeBSD is an excellent choice. Virtualbox 5 has been ported to FreeBSD in a community effort (Oracle does not support FreeBSD hosts). Qemu and Xen are available. And with Bhyve, FreeBSD even has its own modern hypervisor which is a really promising piece of tech! A lot of effort has been put into making the ARM platform a first class citizen with FreeBSD 11 (which is just around the corner now).

The project strives to create a general purpose operating system that is free, open source and under a permissive license. FreeBSD has put a lot of effort in getting rid of GPL’ed software in the base system (you can of course always install it as a package, though!): GCC has been replaced with LLVM/Clang (on major platforms) and a lot of the GNU tools for which no alternative implementation existed have been substituted by newly created, BSD licensed replacements. This is still work in progress, but the plan is to be able to offer a full operating system that is all permissive licensed by the time FreeBSD 12 is released.

FreeBSD’s support for technology like ZFS and DTrace has been mature for quite some time now. Other things like jails have been around like forever and are a stable part of the operating systems literally far more than a decade before “containers” became the thing on Linux. On FreeBSD you can choose between three Firewalls: A modified (e.g. multi-core optimized) variant of the mighty PF, IPFW (FreeBSD’s native firewall) and the old IPF. Also FreeBSD’s Linux emulation is a pretty sophisticated system. Don’t expect it to support all Linux syscalls up to the latest 4.7 kernels – but even though it’s limited to emulating older Linux kernels it still enables you to run some closed source software that is only available for Linux if you need that.

In general FreeBSD is often characterized as “extremely stable” and there’s definitely something to it. However this does not just mean that it’s a rock solid system, mind you! Stability is more than that. E.g. the project follows a design principle called the “POLA”: Policy of least astonishment. You won’t reboot your machine after a major upgrade just to find that your init system has been replaced with something else! You also won’t find that your firewall is inactive because the config format changed and the newer one cannot load your previous rules! It’s not too hard to see the benefit of introducing changes gently over time and with loud heads-up messages instead of quietly. When it comes to supported architectures, FreeBSD concentrates on the more popular ones.

While it’s mostly used with servers, FreeBSD makes a great desktop, too. If you want a desktop FreeBSD, you can of course install vanilla FreeBSD and then add a lot of packages for your favorite desktop environment. However there are two projects which provide a FreeBSD based desktop system (with a graphical installer, packages tuned for desktop usage, and so on): PC-BSD / TrueOS and GhostBSD. The former focuses on a modern OS with a lot of nifty tools – at the cost that it’s available for 64-bit PCs only and forces you to use ZFS. It uses the Qt toolkit and (by default) its own desktop called Lumina. If you prefer the GTK+ side of things, GhostBSD offers the MATE or Xfce desktop environments (and also supports 32-bit systems as well as the older UFS filesystem if e.g. your machine has little RAM). Both are excellent choices if you prefer to start with a complete desktop OS and explore things from there.

OpenBSD

If you choose OpenBSD, there’s a lot of really cool and modern features… that you will have to do without! Oh, don’t get me wrong: That can be a great thing!

The OpenBSD project focuses on security and correctness and for that reason take high code quality extremely serious. They reject the phenomena that is sometimes called featuritis and resist the temptation to implement dozens of “neat” or “nice to have” things at the cost of neglecting the essentials. The OpenBSD developers put quite some effort into maintaining a tidy system. Run across some ancient looking code? Time to ask on the mailing list if somebody does even use that at all. If nobody claims to do so, disable it and wait for someone screaming that it is needed after all. And if there’s any doubt in it still being useful after some time: Bye bye, feature!

Just to give you some examples which prove that those guys really mean it: Not too long Linux emulation has been completely removed because it was old and considered to be in a bad state. Just recently tmpfs was disabled due to a lack of maintenance and may very well be removed in the future as well. And of course it’s security first. Believe it or not: OpenBSD is not afraid to throw things away that pretty much every user of a modern OS takes for granted. Loadable kernel modules can always be a security problem. OpenBSD ripped them out. Usermount turned out to have security issues and – due to its nature – not to be really fixable. It’s gone now.

The project also takes a very tough stance on license matters / politics. There’s GPL(v2) code in the base system but GPLv3 is deemed completely unacceptable. As you would expect from hardliners living up to their creed, OpenBSD still ships with GCC 4.2.1 (the last GPLv2 version) in the base system (you can of course install GPLv3 software as packages)! Even though they heavily patch their compiler, it’s pretty old now and probably is a hassle to continue to maintain. Binary blobs are not accepted even if they’d make something work. Since that would not be debugable or auditable, using blobs would mean a half-assed solution. And OpenBSD follows the principle: “Do it right or don’t do it at all”. They also refuse non-disclosure contracts as a matter of principle. If some vendors choose to be pigheads, OpenBSD usually chooses not to support their hardware. Period.

While most people would probably agree that doing things right is a commendable thing, this may sound a bit too harsh for you. But think again: It’s exactly this seemingly “extremist” stance that brought great benefit to all of us, even if we don’t use OpenBSD. How’s that? The OpenBSD developers have discarded a whole lot of tools because they didn’t live up to their high standards and then took the effort to replace them with a new tool done the OpenBSD way. Tools like OpenSSH, OpenNTPD, OpenSMPD, relayd, spamd, tmux and many more originated in OpenBSD. They even created their own secure webserver (simply called httpd) since they felt that Nginx was getting too bloated over time. As the project has a much smaller community compared to FreeBSD, this really is quite an achievement.

They are also not afraid of taking quite radical steps even if that means a lot of software breaks for them. In 2014 they solved the “2038 issue” that all *nix systems have. Previously OpenBSD pioneered system wide stack protection, ASLR, W^X, etc. All these things are major changes that break a lot of applications. OpenBSD ports maintainers who noticed that some application broke due to changes like these always tried to at least notify the upstream projects or upstream patches if they already resolved the issue. This makes it much easier for other systems to also make these changes since the biggest fallout of doing so was already solved (thanks to OpenBSD).

OpenBSD supports quite a lot of architectures and purposely does so. Even the packages for the most exotic platforms are built on these machines instead of cross-compiling them on faster metal. The idea behind this is that doing so will in some cases expose subtle bugs that would have gone unnoticed otherwise.

While it maintains a rather small base system, OpenBSD still comes with a surprisingly complete set of tools for server duty. Some people say that OpenBSD makes the perfect router and thanks to the PF firewall this may very well be the case. Just don’t be too quick dismissing it for desktop usage! OpenBSD developers are known to be devotedly dogfooding their system on their laptops, servers, desktops – on their everything. If you want to use it with a graphical system, there’s a distribution set called Xenocara on the installation cd. Xenocara is a hardened version of Xorg. Should you ever need help, have a look at the manpages. They are of breathtaking quality.

NetBSD

Alright, NetBSD. This BSD does not have a big community and that community is unfortunately not particularly good at communicating the cool things that they are doing. I’ve used it much less that the two “big” BSDs mentioned before, so I might easily miss quite some information deemed relevant by a regular NetBSD user.

First and foremost NetBSD is known for its portability. Their slogan is “Of course it runs NetBSD!” and if you take a look at the quite impressive list of supported hardware, the project lives up to that. If you get into the greater *BSD community a bit, you’ll sooner or later hit the reference that NetBSD runs on just about every computer and on toasters. Don’t you think this is a joke!😉

Perhaps even more impressive is that NetBSD is just so portable that you can actually compile NetBSD on a Linux system if you want to! And things don’t stop there. Probably the best known project that originated from NetBSD is pkgsrc (pronounced “package source”). If you already have an idea what the ports system on FreeBSD or OpenBSD is, think that plus – of course – a lot of effort added to make it highly portable. Pkgsrc can bootstrap itself on all the BSDs, Linux, Solaris, OSX, Minix, commercial UNICES (AIX, HP-UX, etc) and even Windows (using Cygwin)! What does it do? It provides three things: 1) Portable package tools 2) A portable package creation framework 3) Ports (think package “recipes”).

If you use multiple operating systems you may be tired of having to use different versions of your applications (since all operating systems or – in case of Linux – even all distributions package applications at their own pace). Pkgsrc may be the solution for you since you can use it to build and install the same version of your applications on all of your platforms! Truth be told, not every port builds on all supported operating systems. But still even having 60 – 90% (depending on which software you need and which OSes you run) is quite nice!

But there’s more to it than just portability. NetBSD has been experimenting with a few interesting things over the last few years. For example they’ve imported the small embedded programming language LUA into their kernel. In case you do not know it: LUA is used in many places since it’s very light-weight and easy to use. A lot of games embed it as their scripting language for example. Having this in the kernel could allow for a whole lot of interesting things. However I have to admit that I haven’t looked into this. Same thing for the new firewall, npf, that is part of NetBSD. It’s meant to be very high-performance but I haven’t used it and didn’t hear much about it.

The most interesting concept that I associate with NetBSD is that of the rump kernel, NetBSD’s take on the anykernel concept. If you’re not familiar with that please do some reading yourself. This topic alone would easily deserve its own article – and I’d not be the one to write it since I’m not very knowledgeable in this field.

Dragonfly BSD

Who should try out Dragonfly BSD? Well, you should have a heart for underdogs if you want to try out dfly first. It has by far the smallest community of the four main BSDs and so the chance that you’ll ever run into it in the wild is pretty low. If you don’t mind that (or this even sounds great for you) then there’s nothing wrong with giving Dragonfly BSD a try. In fact from the technical side there are very good reasons to do so!

Dragonfly BSD is the youngest of the major BSDs. It was forked from FreeBSD because its lead developer did not agree with the direction that the FreeBSD project took. Since he was interested in clustering and ultra-high performance computing, these are some of the main aspects that drive the development of Dragonfly.

The Dragonfly BSD project seems to be more interested in creating the best performing OS possible then in things like license matters. While they are looking into making LLVM/Clang their system compiler now, they had no problem importing GCC 5 into their system. For quite some time dfly used NetBSD’s pkgsrc as their official packaging tools. After FreeBSD introduced pkg-ng they switched to using that and a customized ports tree called dports.

Dfly has achieved high performance especially in networking and is the leading BSD when it comes to porting graphics drivers over from Linux. It has dropped 32 bit support and now supports 64 bit machines only. This decision was made due to the little manpower that the project has. A tool called DMA (Dragonfly Mail Agent) has originated from Dragonfly; it is a simple replacement for sendmail on servers that only need to send mail on the local machine.

However the most important reason to give Dragonfly BSD a chance is because it offers HAMMER. HAMMER is a BSD licensed next generation filesystem a bit like ZFS in some regards but different in others. While it lacks some of ZFS’s advanced features it works well even with much less memory and it can even do things that ZFS can’t (fine-grained file history)! HAMMER has been considered stable and production quality on Dragonfly for a while now so this is something to take a look at if you are into filesystems.

Well, and then there’s HAMMER2, version 2 of the HAMMER filesystem with even more features. It is not ready yet and the Dragonfly developers openly admit that it won’t be too soon, but it is interesting enough to keep an eye on (especially since version 1 of HAMMER is proof enough that the team can implement a filesystem that rocks!).

Your choice!

So let me welcome you to the rich and fascinating world of *BSD. You now have a first and very basic impression on what the four main projects “work” like. Try out either one. Try out all of them. Virtualize or use real hardware. The source will always be with you. The choice is, too.

(And please give me feedback about this article and/or tell me what your first encounter felt like after you tried out a BSD!)

Documentation: Linux vs. FreeBSD – a real-world example

With every operating system there comes the time when you need help with something (if you’re not the absolute Über-guru, that is). If you are in need of help, there are many ways to get it. You can ask an experienced colleague or friend if available. If not, you can search the web. There is a very high possibility of the information that you need being out there, somewhere. If not, you could ask for help and hope that somebody answers. Well, or you could consult the documentation!

In most cases somebody has been right there before and asked for help on the net and somebody else gave an answer. That answer may or may not be correct, of course. And in fact it might even have been correct at some point in time but is no longer valid. This is a very common thing and we have learned to optimize our searches to more or less quickly find the answers that we need. After getting used to that habit, “google it” (replace with $search_engine if you – like me – try to avoid using Google services when possible) is probably the most common way to deal with it once you hit a problem on unfamiliar ground. So while users of Unix-like systems are usually aware of the existence of manpages, I’d say that especially younger people tend to avoid them. And really: You don’t need them. Except for when you do!

Public WLAN

Last week I had two appointments in another city. So I took one day off from work, got up early in the morning and drove about 1.5 hours to the first one. The second one was a few hours later and so I was left with something extremely precious: Free time! To make it even better, neither my children nor my wife were around. The perfect opportunity to get something done!

One of my hobbies aside from computer stuff is writing. In addition to shorter stuff, I also have a fantasy novel (called “Albsturm”) that I’m writing on as time permits (which it hardly ever did during the last two years). And so I figured that it would be a good idea to take a laptop computer with me and spend some hours writing (hint: Like always, I didn’t write a single sentence!). I have two reasonably new laptops that I could choose from, one running Arch Linux, the other one FreeBSD and OpenBSD. The latter is the smaller one and for that simple reason I took that one with me.

It was a warm day and I decided to sit down at a café, have a drink and do my stuff there. When I found one, I saw a sticker which told me that public WLAN was available there. Hm. Other than writing I also had a more or less urgent email to write. Should be a quicky, just a few lines. So I thought that I should probably start with that.

Offline!

The only problem was that I had no idea whatsoever on how to connect to the WLAN using FreeBSD or OpenBSD! In fact I had no idea how to do it on Linux, either. I’m an “all cable guy”. It feels like about two decades ago that I had my first wireless mouse. I really liked it – until the batteries ran out of charge in a very bad moment and I didn’t have any replacement ready. Wireless stuff may be convenient as long as it works, but I prefer reliability over that. And I also like to set up basic things once (which means that I wouldn’t like to have to change a WLAN channel if my neighbor gets a new access point which occupies the same one that I had used before – stuff like that).

The three or four times that I had used WLAN before was on a Linux box using the graphical Network Manager which does all the magic behind the scenes. Yes, I’m aware that PC-BSD has its own tool which does the same job and GhostBSD has another for people like me who prefer a GTK application over a Qt one. I had neither PC-BSD nor GhostBSD on my laptop however. Just vanilla FreeBSD (with EDE as the desktop) and OpenBSD without any desktop (because I didn’t have time to install one, yet).

So there I was, offline and looking for a way to go online. Obviously “google it!” or some variant of that did not apply here. Sure, the adventure could have ended just there. But I am a weirdo who refuses to take a mobile with him everywhere he goes like most other people seem to do these days. Now if that’s shocking for you or you just cannot believe that someone who deals with tech does not have his mobile in reach all the time: Just imagine that I had one but it ran out of power (I’ve seen this happen to friends often enough to know that it’s quite common)!😉

Ok, what now? Thinking about it for a second, I realized that I had made a mistake when installing my system. You don’t install doc when you’re setting up a new system, right? The (absolutely excellent) FreeBSD handbook is available online after all. So why should you? Yeah. So am I on my own here? No! It’s me and a man’s man(1)! Will that suffice to go online?

Help!

Thanks to my previous exposure to help systems, this was the moment where I could have felt a cold chill (which would actually have felt good due to the warm weather). Remember the Windows 9.x “help” system? I cannot remember a single time when it had actually helped me. It either found nothing even remotely connected to my problem or it gave some generic advice like “ask the network administrator” (I AM the “network administrator”, dammit! I’m the guy who plugged those four cables into the switch and gave static IPs to the PCs!). It was utterly useless – and in a later version they “improved” their help by adding a stupid yellow dog… (When PC people talk about “the good old times” this is what you should remind them of :p)

But let’s not waste any more time on the horrible demons of the past and skip to the friendly daemons of today! I’ve used manpages a few times on Linux systems. This was a much better experience but still a vain effort often enough. The worst thing: For a lot of commands there are both a manpage and an info page – and those two are not identical at all! With a bit of bad luck you skimmed through one help text but the relevant information is only present in the other. Even though I can see the limitations of the older manpage system and understand the intent to create something better… No, sorry. If GNU really wanted to go with info pages instead of manpages they should just have created manpages which point the reader at the info page for each command. Just don’t make me read both because they have different information in them!

FreeBSD has a natural advantage here due to its whole-system approach. If you install third-party packages (say GNU’s coreutils) you will be in for the same mess. But everything that belongs to the base system (and that’s a whole lot of stuff!) is a consistent effort down to the manpages. And from what you hear or read on the net, the BSDs pride themselves in dedicating a fair amount of time to write documentation that’s actually useful! Does the result live up to that claim? We’ll see.

Where to start?

Manpages… Ok, sure. Just what should I start to look for? As I said, I didn’t know too much about the topic. Hm! I couldn’t think of anything quickly, so I actually did a apropos wlan. It wasn’t a serious search and I didn’t really expect anything to show up. Here’s the output of that command from a Linux box:

apropos: nothing appropriate

So was I right there? No! I was in for a first pleasant surprise. Here’s the output on my FreeBSD machine:

snmp_wlan(3) - wireless networking module for bsnmpd 1
wlan(4) - generic 802.11 link-layer support
wlan_acl(4) - MAC-based ACL support for 802.11 devices
wlan_amrr(4) - AMRR rate adaptation algorithm support for 802.11 devices
wlan_ccmp(4) - AES-CCMP crypto support for 802.11 devices
wlan_tkip(4) - TKIP and Michael crypto support for 802.11 devices
wlan_wep(4) - WEP crypto support for 802.11 devices
wlan_xauth(4) - External authenticator support for 802.11 devices
wlandebug(8) - set/query 802.11 wireless debugging messages

Not bad, huh? 9 hits compared to… 0! I had nowhere better to go, so I read wlan. It provided a fair amount insight into things that I was not too interested in at that moment. But it had a rather big SEE ALSO section (which I feel kind of lacking in the Linux manpages that I’ve read so far). This proved extremely useful since a lot of device drivers were mentioned there and I figured that this would actually be a good place to really start.

Dmesg told me that my machine has an “Intel Centrino Advanced-N 6205” and that the corresponding driver was iwn. However ifconfig showed no iwn0 interface. There were only em0 and lo0 there. How’s that? I figured that it probably had to be set up somehow. And had I not just read about the generic wlan driver?

The wlan module is required by all native 802.11 drivers

The same manpage also pointed me to ifconfig(8) which makes sense if you want to do interface related stuff (unless you’re on newer Linux systems which sometimes do not even have ifconfig and you have to use the ip utils).

The ifconfig(8) manpage is a really detailed document that helped me a lot. So it’s only

ifconfig wlan0 create wlandev iwn0

and my wlan interface appears in the list showed by running just ifconfig! That was pretty easy for something which I would have never figured out by myself.

Let’s go on!

The first step of what could have been a painful search turned out to be so surprising easy that I was in a real light mood. So instead of just getting things to work somehow(tm), I decided to do it right instead. It was only one simple command so far but I wouldn’t want to enter it again after each reboot. So it was time to find out how to have the init system taking care of creating my interface during system startup.

Phew. That could be a tough one. What obscure configuration file (or worse: systemd “unit file”) could WLAN configuration stuff go into? Hey, this is FreeBSD! Want init to do something for you? Have a look at /etc/rc.conf!

Hm. Sometimes configuration files have their own manpages, right? But even if there was one, it could hardly cover everything. Would somebody take the time and put what I need in there? Ah, let’s just give it a shot and man 5 rc.conf. Yes, there’s a manpage for it. But not just a manpage. I mean… Wow, just wow. I’m still amazed by the level of detail everything is described with! Should you ever take a look, you’ll be in for a treat of over 2400 lines! Does it cover WLAN interface creation? You bet it does! And it holds more information about that topic than fits on one terminal screen. In my case it boils down to:

wlans_iwn0="wlan0"

Really simple again – which is really encouraging if you’re new to a topic (on an operating system you’re only slowly getting familiar with because you have to spend most of your time with Linux machines).

The manpage also mentions wpa_supplicant(8) and after reading a bit about it and wpa_supplicant.conf(5), I had my system automatically make a WLAN connection during startup: It received and ack’d a DHCP offer and got an IP. Great!

Hold captive by the portal

Time to fire up a browser and surf to some website to see if it works… Oh my. What’s that? A captive portal redirects me to a page with payment information! That’s not quite what I’d call “free WLAN”! What happened? The page says “Telekom” but the sticker said that the hotspot was provided by another company. So I must be connected to the wrong one…

So it’s reading ifconfig(8)’s manpage again. Turns out that ifconfig wlan0 scan returns a list of available networks. So far so good. Of course the manpage also explains how to manually connect to a network of your choice. But this is where things went into the wrong direction for me.

The SSID of the network that I wanted to connect to was too long to fit into the column that ifconfig reserves for the output… Gosh. Now how would I connect to that one? Guess the rest of the name? Probably not a good tactic. What else? I could not connect to the network that I knew was free and I didn’t want to just randomly try connecting to the others.

Autoconnection makes its decision by signal strength. It’s rather unfortunate that the stupid paid Telekom one had a better signal where I was sitting. But by blocking that one network there’d be a good chance that the right one would have the second best signal, right? So I only have to somehow blacklist the Telekom network.

A long story short: This proved to be a dead-end. I still have no idea if it’s even possible to blacklist a network using wpa_supplicant.conf. It probably isn’t and the only way to go is define the desired network with a higher priority than the undesired one. It took me quite some time to give up on this path that seemed to lead nowhere.

What now? It looked like I had to somehow get my hands on the complete SSID of the right network. But how to do that? I could of course always ask a waitress as she probably either knows it or at least could ask somebody who does. However after spending quite some time on the matter, I wanted to figure it out myself. Finally I came across ifconfig(8) again and it mentions the -v flag to show the long SSIDs (and some more info)!

With the full SSID known to me I could adjust my wpa_supplicant.conf – and after a reboot the system picked the right network. My browser was lead to a different captive portal and after I read and accepted the license terms (which were quite reasonable), I was free to surf wherever I wanted.

I quickly wrote and sent the mail that lead me to this adventure in the first place. Then I shutdown -p now my system, put the laptop in my car and drove to my second appointment.

Summary

I’ve had some Linux experience for almost two decades now and used it on a daily basis for about half of that time. In contrast to that I’m really new to the BSDs, seriously using FreeBSD for less than a year. I probably know less than 1% of the common taks on that OS – and even less on the topic of WLAN which I avoided as far as I could.

Getting my laptop to connect to the net via WLAN in a café using just the manpages because I was offline until I reached my goal seemed like a painful adventure full of potential pitfalls. Instead it proved to be an unexpectedly pleasant ride in unfamiliar territory.

There are many sources on the net that say BSD has far superior documentation compared to Linux. And I was impressed enough about that fact to add another one by writing this post. So if you’re a *BSD user and you need help I can only give the advice to take the time to read some manpages instead of looking or even asking on the net. It is much more rewarding to figure out things yourself using the documentation and the chance is quite high that you’ll learn another useful thing or two from it!

Can the same thing (connecting to a WLAN without graphical tools) be done on Linux? Certainly. How would you do that? I have no idea. Is there an easy way to figure things out just using the manpages? I kind of doubt it. With a lot more time on your hands: Probably. But after learning what real documentation tastes like, I don’t feel like trying it right now. I may do it in the future to complete the comparison. Or maybe not.

Version control (pt. 2): Generations and intended use

In the previous post I gave a little introduction to Version control, explaining a few basics that help to understand the topic. This post assumes you know these things that version control systems have in common. So now it’s time to discuss what sets them apart – not individually, yet, but in terms of characteristics some of them share with each other.

Generations

Each version control system (VCS) has its unique advantages and disadvantages. However there are some traits which are common to several of them. And since the programs that share those traits were typically released rather close to each other, it makes sense to speak of generations of version control systems.

So far there are three of them with the first obviously being the oldest and the third generation the newest. What may surprise you is the fact that the earlier versions, even though being much more limited, have not completely disappeared. How come? Well, just keep it in mind while we take a look at those generations. Perhaps you can see where tools of older generations may still make sense!

The previous article discussed manual “version control” and its limitations. In short: It can work for you if your project is rather small, you’re working on it alone and you’ve got the discipline to always do proper file backup before you make bigger changes. In a world of more sophisticated software it’s quite unlikely that many projects meet all of those requirements. Therefore it totally makes sense to develop programs that would assist you in doing proper version control on your projects.

The first generation

The first generation did exactly that: It preserved any changes you made to a single file. Yes, it is one characteristic trait of the first generation that it works on a per-file basis. Version control is entirely separate for each and every file that you choose to record changes for. For each file it manages, the VCS creates a “history file” which contains all the differences from one version to the next plus a comment.

Originally, it was common to use VCS of the first generation on multiuser systems. If multiple users can work on the same project at the same time (via different logins), it’s quite possible that conflicts arise. If two persons make changes to the same file, the one who saves last “wins” – overwriting all changes that somebody else may have made in the meantime. To avoid that, locking was invented. Files can be locked while they are being edited. In case somebody decides to edit a file, causes a lock and then does something else, the file remains locked. An administrator can however break a lock if something like this happens.

Today VCS of the first generation are more or less obsolete. There are niches where they managed to survive and are still being used. One example is management of configuration files on *nix systems without centralized configuration management. These configuration files are only relevant to the system they exist on so that missing networking capabilities (which the second generation introduced) do not mean any disadvantage. And most of the time these configuration files are separate entities not related to other files and for that reason not even the limitation of only managing a single file is a problem here.

The second generation

There are two important aspects that set tools of the second generation apart from those of the first: They offer network capabilities and they can manage multiple files in one project! The later ability solves a whole bunch of problems which made 1st generation tools hard to work with on anything but very small projects. Managing each file separately does not sound too bad at first. But think about it for a minute.

Let’s imagine, we work on a simple project. Nothing too fancy: A few source code files, one header file. Currently the program is broken and we decided to go back to a working version. Good thing that we have version control, right? Right… Sort of. The file main.c is currently at revision 96, foo.c at revision 44, bar.c at 24 and baz.h at revision 7. See the problem? After we found out that revision 89 broke the program and we reverted back to 88, how do we find out which revision number of the other files belongs to revision 88 of main.c? Yes, we have time stamps and we can find out which revisions all of our files had when the program was working when the main file was at revision 88. Maybe it’s not even that bad when we only have four files. But what if we have 20? 100? It’s cumbersome and really a waste of time. Keep things like this in mind and you’ll definitely come to appreciate the ability to manage multiple files together in one project where the revision number increases whatever file was changed and however many files were modified!

Now that larger projects are possible because the whole project is managed together in one repository, it makes sense to use the network as well. This networking capability is achieved by providing one centralized repository which all project members (or even everybody interested in the project) can checkout to create a local working copy of the latest revision (or any older one if needed). Changes can be made locally and after committing them they are checked in back into the centralized repository. Since the tools of the second generation will only allow checking in if nobody else did a check-in in the meantime (if somebody did, you need to update your working copy first and merge your and the remote changes if at least one file was modified by both) locking is also not necessary anymore!

Today tools of the second generation still play an important role. Their attractivity is declining, however. This is due to a few shortcomings which the third generation tries to address.

The third generation

The big innovation that is common to all tools of the third generation is that they work decentralized. Users usually don’t checkout files from a central (probably remote) repository. Instead they clone the full repository and then checkout the files from their local clone. Since the local repository is exactly the same as the original one, there’s no longer one central repository – at least from a technical view. And while cloning requires to transfer a lot more data over the wire (especially for large projects), there are some huge benefits to it.

If you have a local clone, you can work on the project even when you’re not online. You can see the complete history and checkout earlier revisions if you need to – all without having to access a central repository. If you’re online, you can always sync your local clone with the original repo (pull down changes) or even the original one with your local repository (push up changes) if you have write access.

One of the biggest advantages of decentralized tools is that they make forking much easier. While forking a project has been something not well liked in the past, you’ll often see projects asking you to fork and play with their code today (e.g. the well-known “fork me on github”). Experience has shown that quite some people fork a project, add a feature they need and then give their code back to the project (this is done by creating a pull request which invites the administrators of the original project to pull in the changes if they want).

Which one to use?

If you like software history, there’s nothing wrong with trying out tools of the older generations. But if you’re just starting out with version control and you want to learn something now, it makes sense to choose a tool of the third generation. Which one would I recommend? I can only give the usual answer to such a question: It depends. Each one has its strengths and weaknesses. In the next blog posts we’ll take a closer look at some of the open source VCS of all generations. This might help you to choose the right one for your purpose.

Version control (pt. 1): An introduction

This post is the first part of a series on version control. It provides an introduction by explaining what that actually is, why you should probably use it and how it works in general.

Important terms

Version control (also: revision control) is a means of preserving various versions of a file or of multiple files. This can be done in a lot of different ways but over time some best-practices have emerged that are more or less followed in all modern version control systems.

There are lots of cases where version control makes sense. One of the most common ones is software development where using version control is virtually mandatory. That is why there’s even a separate term describing this form of version control: source (code) control or source code management.

We can group various version control systems together in two groups: local as well as network-based systems. The latter can be further differentiated between centralized and distributed ones.

In short: Why you may need it

Depending on the choice of the tool there can be various situation where you can benefit from version control:

  • Version control enables you to quickly return to an older, known-good state in case something broke
  • Version control gives you the possibility to precisely document any changes you made and thus provides both tractability and a quick overview of changes
  • Version control solves a lot of the problems that occur if multiple people work on the same files at the same time
  • Version control makes it simple to clearly find out the author of any change

Or as a former colleague of mine put it (in a very vivid way):
Why to use version control!

Manual version control

The simplest form of version control is working with a backup copy. You make a copy of e.g. a configuration file before making a change to the live file. Afterwards you test your changes and if they seem to work you either delete the backup or keep it for reference. If the changes had undesired effects, the original is overwritten with the backup (and the latter usually deleted again). This is actually a (rather primitive but sometimes sufficient) form of version control: Thanks to the backup copy you have two versions of the file at your hands!

Backup copy – we’ve all done it

Another variant is to make a copy after a fixed amount of time (or at random). Often people prepend the date to the file name. If all you want to accomplish is that e.g. the data as it was at of the first of each month is preserved for one year, that’s also a sufficient method (along with rotating the backup copies so that you don’t keep around more of them than you need).

Those means of manual version control are however pretty limited. And worse: There’s plenty of room for making mistakes!

Manual version control

How a version control system (VCS) works

Let’s think about a simplified versioning process by pretending to do things by hand. For each recorded change of a file you’d make a copy of the file and keep all the old versions of it. A VCS does not forget a single version of a file it monitors! That’s what it is meant to do, after all. Each new version of the file gets a comment that is meant to briefly sum up the changes that were made.

Keeping probably dozens or even hundreds of files around because one file has that many revisions, would really clutter your disk. Also it would not make sense to keep nearly the same file twice if only one line was changed! That’s why local VCS which also version on a per-file base, keep a “history file” around for each versioned file. That file records only the changes between the various versions as well as the comments.

Network-based VCS are able to organize a whole project (multiple files together instead of each one separately). They also record the changes instead of the full files for each revision as well as the comments. All of that data is collected in a so-called repository.

If a new team member wants to start working on the project, he or she first needs to get the files of that project. For centralized VCS this is done by checking out the most current revisions of all the project files from the remote repository. By doing so, a local working copy of the project files is created to be worked on by the user. When using a distributed VCS the remote repository is cloned instead (thus receiving the full repository with all revisions and not just the most current version of each file). The working copy is then checked out from the local repository clone.

At the beginning of a new project there is no repository, yet. In this case either an empty repository is created, checked out and the new working directory is populated with files. In a next step, those files are placed under version control (which means that the VCS is told to watch them and record changes). Then all changes (all of the files since they are all new right now) are placed into the repository by doing a commit.

After every change made to the project you do a commit again, recording the changed state inside the repository. Other project members can now get a current copy from the repository. This way it’s easy to work together on the same project without the risk of (unknowingly) get into the way of somebody else.

OpenBSD/FreeBSD (ZFS) dual-boot & thoughts about GPT/EFI

In the previous post I wrote about how to get a computer up and running with a dual-boot of FreeBSD and OpenBSD while using full disk encryption. This worked quite well but a bit later I decided that it would be a good time to do the FreeBSD installation again – this time going the modern way and using ZFS on root. This has led to a surprisingly large amount of problems. In the end I got a working system that uses ZFS, so that I’ve actually got an alternative howto for FreeBSD (the OpenBSD part remains the same as before and can be looked up in the previous post). But it’s all a bit different from what I first thought it would be.

Doing things today’s way: GPT

GPT (GUID Partition Table) is a more modern partitioning scheme meant to replace the old MBR style partitioning, redeeming us from limitations we had to live with in the past. It can deal with drives up to 8 Zettabyte (10²¹ Bytes) instead of being limited to 2 Terabyte (10¹² Bytes). That limitation used to be no problem and it still isn’t too bad for most home users since drives bigger than 2 TB are not that cheap, yet. But it obviously won’t be too long before this will be a common issue.

OpenBSD bootloader chainloaded by boot0

A more serious limitation is that MBR only supports 4 partitions. Yes, we all used to call these “primary partitions” and made use of “extended partitions” nested inside one of them to get around that limit. BSD users created disklabels instead to embed more partitions inside disk slices (“MBR partitions”). With GPT this is trouble of the past and you are free to create just about as many partitions as you think you need. There’s no need for embedding BSD disklabels or doing any MBR trickery – which is nice.

Also GPT supports naming partitions. So by using them we can do away with the old glabel mechanism that FreeBSD offers and use native GPT labeling instead. That advantage comes with a little disadvantage, though. By default FreeBSD’s gpart shows such partitions multiple times: Once by their label and once by GPT-id. This can be a bit confusing as you first come to the GPT world. Fortunately there’s a simple solution: Setting a sysctl you can simply disable GTP-ids if you opt for labels (which you should since they are much more meaningful).

GPT is a requirement for machines that use EFI booting. The good news is that all recent x86 hardware comes with EFI (and thus GPT) support. The bad news is that while some machines do support GPT when booting from EFI, they don’t if you choose to boot from legacy BIOS emulation. So if you want to stick to BIOS you may want to check if it is capable of booting a system on a GPT partition. My EliteBook 8470p is fine with GPT partitions in BIOS mode. So I was good to give GPT a try.

Problems with GPT

FreeBSD works great with GPT (and has done so for quite some time now) so there’s nothing wrong installing it using GPT partitions. The first obstacle that I encountered was the boot manager. FreeBSD comes with the nice and simple boot0 tool that I used in my previous howto. Too bad that it’s MBR only! So to continue down the path with GPT, it’s using another boot manager. Usually GRUB is used for that purpose. And while I would certainly prefer to go without a boot manager that has “grand” in its name, I think that using it is acceptable.

The FreeBSD bootloader

A bigger problem awaits, however. OpenBSD didn’t support GPT prior to the current version 5.9. Since it does now, everything should be fine, right? Wrong. If I didn’t miss anything, OpenBSD supports GPT only in conjunction with EFI booting! I did not find a way to use GPT partitions with OpenBSD in BIOS mode. Should anybody have more information on this, I’d very much like to know if it either is possible or if support for this is planned for the future.

Choosing EFI?

This means that not too far down the road there’s already a solid blocker. I had next to no knowledge about the EFI complex and I successfully avoided that topic in the past. So my two options to go on were to either give up on GPT and simply stick with MBR or to make the bold move and go for EFI. I would prefer to try out things one after another but meh… I decided to read a bit about the basics of EFI and then give it a try. To be honest, I don’t like what I read too much. Sure, there are quite a few interesting things that EFI can do. But then again I do not believe in unneccessary complexity. And above all: I don’t like the security implications of it. Not a bit. Trust is not something that I give away for free. Trust has to be earned. Unfortunately closed source vendors have done very little in the past that makes me think I want to trust them. Anyway… EFI is the (near?) future and it will not be easy to avoid it altogether in the next few years.

Alright. So I turned off legacy BIOS emulation on my machine and booted FreeBSD (there are separate EFI images for FreeBSD 10.3 – make sure you pick one of those if you want to go with EFI!) into the installer. Everything worked smoothly and after a couple of minutes my new FreeBSD system had booted. It was so simple and dull that there’s not too much I could write about (which is a good thing since everything really just worked).

Next step: Installing OpenBSD. I read that the memstick image supports EFI booting and I can confirm that installing works just as well as with FreeBSD. Again the new system works like a charm – the OpenBSD people have obviously done a good job for 5.9. OpenBSD was able to cope with GPT just like you would expect.

FreeBSD desktop: EDE, pcmanfm, terminator, smplayer

So far everything looked good. Final step: Make the computer offer a means of booting either system! Honestly I did not expect that step to be a show-stopper. It turned out that it is. Most people seem to use the EFI boot manager rEFInd. You can install it from Windows, OS X and Linux. Yes, that’s it. Now, I was not surprised that it was not ported to OpenBSD. But it did surprise me that it’s not available for FreeBSD!

It may be a quite simple thing to toss in a Linux CD and install an EFI bootloader – or maybe not. I have no clue if there are any other obstacles waiting. But this was the point where I stopped. I wanted a dual-boot BSD system and I don’t want to have to use Linux to do that. I’ve got my pride, too, you know. :p No seriously, at that point I decided to give up EFI for now and continue another time. Maybe that will be worth its own blog post. Who knows.

Back to BIOS / MBR… for now

Here we are, back to using BIOS / MBR. I wanted to use a purely ZFS setup for FreeBSD but another problem showed up: The bootcode to load a kernel from ZFS is quite a bit more complicated than its UFS equivalent. For that reason it doesn’t fit into the boot sector of a partition but needs its own small partition instead. However with gpart the type freebsd-boot seems to be only supported for GPT…

To get anything up and running again (for the time being my on-call laptop was broken after all and my next duty was around the corner!) I settled with a UFS boot partition for the unencrypted /boot partition.

Manual install: FreeBSD with ZFS, GELI

The auto_ashift adjustment is needed for ZFS to adhere the 4k alignment. And to be able to set that sysctl, the ZFS kernel module has to be loaded. That’s a little detail but it took me a moment to figure out what’s going on. Everybody advices to set the sysctl but the installer kept on telling me that there was no such sysctl… Which makes sense once you know that ZFS is not loaded into the kernel by default.

For the layout of the datasets I followed the defaults as used by the automated root-on-ZFS installation (zpool history is a very interesting command! If you don’t know it – try it!). I just changed the order so that similar datasets follow each other which avoids a bit of typing by bringing back the previous command and editing it.

It’s more likely than not that this is not the best way to do things. But it does work. Any suggestions or other comments are welcome of course!


In the partitioning shell:
tcsh
dd if=/dev/zero of=/dev/ada0 bs=1m
gpart create -s mbr ada0
gpart add -a 4k -t freebsd -s 98G ada0
gpart add -a 4k -t freebsd ada0
gpart create -s bsd ada0s1
gpart bootcode -b /boot/boot0 ada0
gpart bootcode -b /boot/boot ada0s1
gpart set -a active -i 1 ada0
gpart add -t freebsd-ufs -s 2G ada0s1
gpart add -t freebsd-swap -s 4G ada0s1
gpart add -t freebsd-zfs ada0s1
glabel label clear /dev/ada0s1a
glabel label swap /dev/ada0s1b
glabel label system /dev/ada0s1d
newfs /dev/label/clear
dd if=/dev/random of=/dev/label/system bs=1m
geli init -b -s 4096 -l 256 /dev/label/system
geli attach /dev/label/system
kldload zfs
sysctl vfs.zfs.min_auto_ashift=12
zpool create -o altroot=/mnt -O compress=lz4 -O \
atime=off -m none -f zroot /dev/label/system.eli
zfs create -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ zroot/ROOT/default
zfs create -o mountpoint=/usr -o canmount=off zroot/usr
zfs create -o mountpoint=/var -o canmount=off zroot/var
zfs create -o mountpoint=/tmp -o exec=on -o setuid=off \
zroot/tmp
zfs create zroot/usr/home
zfs create zroot/usr/src
zfs create -o setuid=off zroot/usr/ports
zfs create -o setuid=off zroot/var/tmp
zfs create -o exec=off -o setuid=off zroot/var/audit
zfs create -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/log
zfs create -o atime=on zroot/var/mail
zfs set mountpoint=/zroot zroot
exit
exit

In the "final modifications" chroot:
mkdir /realboot
mount /dev/label/clear /realboot
mv /boot /realboot
ln -s /realboot/boot /boot
echo 'geom_eli_load="YES"' >> /boot/loader.conf
echo 'zfs_load="YES"' >> /boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot/ROOT/default"' > \
/boot/loader.conf
echo '/dev/label/swap.eli none swap sw 0 0' >> \
/etc/fstab
echo '/dev/label/clear /realboot ufs rw 1 1' >> \
/etc/fstab
sysrc zfs_enable="YES"

Setting up a FreeBSD/OpenBSD dual-boot with full disk encryption

A bit over a month ago, I bought my first refurbished laptop. Previously I used a ThinkPad (owned by the company I work for) for on-call duty. It’s running a Linux distro which would not be my first choice at all, it has a small screen and – it’s not my property. I wanted my own laptop and since we’re allowed to use whatever distro we prefer, I thought that I’d be going with Arch.

(I you’re just interested in the commands to enter, have a look at the end of this post where I put a list of them.)

*BSD in production

On a second thought: Why not use *BSD? For me it would mean going to use a *BSD desktop “in production” after only running it privately. Thanks to the great BSDNow! show I feel confident enough now to give it a try. The company that I work for is running some FreeBSD servers, too, so it’s not something entirely strange and unknown. I went with asking if using BSD for on-call was ok. The answer was what I expected: If I thought that it would work ok I should well try it. The only requirement was that I’d encrypt the disk (the same rule would apply to Linux, too, of course).

Next question: Which BSD to use? Since I’m just getting into *BSD, I’m not really familiar with all of them now. Net and Dragonfly would certainly be interesting, but since I need that box for work that’s not an option. I need something that I know enough to be able to work with. Of course it would be best if I could learn something at the same time… So, what’s the best way to learn more? Probably tracking -CURRENT! But what if something breaks? I cannot afford that. And which BSD to use anyway? I work with some FreeBSD servers, so more in-depth FreeBSD knowledge would make sense. Then again I’ve really come to like Puffy and all he stands for…

That would be a hard decision! Finally I decided not to decide – and to just install both instead. This also has the advantage of having a second system if either CURRENT should ever break!

Hardware: HP EliteBook

I bought an HP EliteBook 8470p. Why didn’t I go with Lenovo even though those are known to work best with *BSD and I obviously need something that seriously works? Well, there’s one reason for me: With the ThinkPads keyboards just totally suck. I have no idea who came up with that sad story of “Hey, let’s just put the Fn key where Ctrl belongs and vice versa!”. No idea whatsoever. But I know for sure that it drives me insane. No fun at all when you’re working on-call at four AM, barely awake, and nothing happens when you have to CTRL-C something quickly. I could never get used to it ever!

So for that very reason it had to be some other hardware. I had this older HP laptop that a friend sold me for a few bucks a while ago. I can’t remember which model exactly and cannot look it up since I don’t have it anymore. (When my mother’s old computer died as I was over on a visit, my father thought about replacing it with a Windows box since that’s the only thing that he knows. To avoid that, I set up said old HP laptop that I had with me as a replacement and gave it to her. She’s been using it happily since.) That laptop had been a pleasant experience when I had OpenBSD on it and so I decided to give that EliteBook a try.

It works fairly well for most things. On FreeBSD there was the problem with the Intel video driver but since I’m running 11-CURRENT video is all working great even when I quit X11. WiFi is detected according to dmesg but for some reason no iwn0 shows up if I run ifconfig. I didn’t have time to look into that further, however. On OpenBSD backlight gets turned off if I quit X and thus the screen is a bit dark then. Since I usually quit X to shut down the computer afterwards, anyway, that’s only a minor issue. WiFi is correctly detected and I confirmed it to work. Suspend works when I close the laptop but when it wakes up the keyboard does not work anymore. These are the only issues that I ran into so far.

What is the exact use case?

FreeBSD can use ZFS while OpenBSD cannot. I’m not sure if FreeBSD’s and OpenBSD’s UFS/FFS filesystems are compatible (I think OpenBSD’s implementation misses quite some of the newer features). The encryption methods used by the systems however are definitely not compatible. So it doesn’t matter anyway in this case and I’m free to choose whichever filesystem I want.

Since I’ll be compiling FreeBSD-CURRENT now and then (and in general plan to do some stuff that likes much memory to be available), I decided to go with UFS. Yes, there are scenarios where ZFS is simply overkill! There’s only one drive in the laptop, it’s not extremely big and it won’t hold any important data. I have no need any particular ZFS feature on that system, so going with UFS should be fine. (That plus the fact that I’m still reading Lucas’ and Jude’s excellent book on ZFS and intend to play with that filesystem on another machine)

Prior to version 5.9 (released after I originally wrote this), OpenBSD only really supported the MBR partitioning scheme so going with that was an easy choice. I’ll stick to it for now because I need some time to play with it first. I’m going to do everything again in a VM so I can take screenshots for this article.

Installing FreeBSD

The installation begins just like an ordinary FreeBSD install: Boot up the installer media and make your way through the setup questions. When the installer asks about the partitioning however, we’re going to do that by hand.

Choosing to partition by hand

The pure bourne shell is not very comfortable for interactive use, so it generally makes sense to use a more advanced shell (like tcsh) for convenience features like auto-completion. Should you not know which drives your machine has, camcontrol can help you. If you want to start with a clean drive, you can zero out everything with dd (when I bought my laptop it had Windows 7 on it that I wanted to get rid of).

Zeroing out the disk

If you’re not familiar with what partitions and slices are, you may want to have a look at an older post where I wrote up a little excursion about that topic.

First an MBR is created and then two slices are added to it. The first one gets 100 gigabytes, the other the rest (which is also 100 GB in my case). Both slices are created aligned to 4k sector size of the hard drive. Then a BSD disklabel is added in the first slice. After that, boot0 (a simple boot manager) is written on the drive and the standard bootcode into the first slice. Finally the first slice is marked as active for booting.

Slicing the disk

Now three partitions are created inside the BSD label: One for boot (which will hold the kernel and cannot be encrypted), one for swap and one for the system (which will be encrypted). Glabel is used to give these partitions a more meaningful name than ada0s1a and the like. Since the system partition will be encrypted, it makes sense to write some garbage all across it so that it is impossible to see which part holds data and which does not. This takes quite a while and you could of course skip this. As long as your patience lives up to paranoia, that little bit of extra security is worth the wait!

Creating and labeling BSD partitions

Next the system partition is initialized with GELI, one of FreeBSD’s two military grade encryption methods. I only use a passphrase to unlock but you can also use a key (or both) if you wish. After attaching the new GELI partition, a new GEOM provider, system.eli is available with the clear data for you (and your programs) to use.

Creating and attaching the GELI partition

Now it’s time to format the two data partitions (the swap partition does not need any formating). You could also use journaling UFS for the boot partition but it’s usually not necessary.

Creating filesystems

Copy over the boot directory and add two lines to loader.conf so that you’ll have the chance to unlock your GELI partition during system startup. What remains is writing a fstab. Notice that for some reason I’ve forgotten to put swap.eli in there on my screenshot (even though that’s what I have in my script). What this does is using a one-time key for your swap on each boot, thus making sure that any data that remains on the swap partition is useless once the system was powered down once. You do not have to initialize GELI for this. FreeBSD knows what to do when it sees swap.eli.

Mount the decrypted system partition on /mnt as that’s where the installer expects it. And don’t forget to create the clear directory as we demand that in fstab and the system would not boot up correctly if it was missing. Then exit the shell and continue with the installer.

Copying over /boot and writing loader.conf and fstab

Once the installation has finished, the installer will ask you if you wish to make any final modifications. Answer yes and it will drop you into a shell in a chroot of your new system. Delete /boot (that directory lives on the encrypted system partition and the bootloader could not find the kernel there anyway) and make it a symlink pointing to /clear/boot instead. This step is not actually required. But if you don’t do it, you won’t be able to update your system the normal way. If you want to only mount the real /boot by hand whenever you upgrade, that’s fine, too, of course.

Chosing to make final modifications

Exit the shell, reboot and remove the boot media. Then reboot. Your boot manager (boot0) will offer you two FreeBSD systems. Hit F1 to boot up FreeBSD. Don’t hit F2. There’s no system there, yet.

Installing OpenBSD

The OpenBSD installer is neither pretty nor does it offer any kind of menu system. However it is simple, effective and straight-forward. Choose to install OpenBSD, set your keymap, enter a hostname, configure the net and set a root password.

Hostname, network and password configuration

Choose to run an SSH server by default, whether to prepare the system for X11, if you want the display manager XDM to be started automatically. Create a user now or do so later. When asked for the timezone, give a ! instead to drop into a shell.

Going to a shell

If you don’t know your disks, look inside the dmesg for the name. Now use fdisk to change the type of the second partition from A5 (FreeBSD) to A6 (OpenBSD). Then use disklabel to create a swap partition and a main partition. Make absolutely sure that the later has the type RAID!

Partitioning for OpenBSD

Encrypt the new softraid with bioctl then exit the shell. Now enter the correct timezone and choose the newly created softraid for the installation! Dedicate the whole softraid disk to OpenBSD but edit the partitions to fit your need. You do not need a swap partition on the softraid because we created a separat one on the real disk, remember? For that reason, after OpenBSD formated the partitions you created, the installer will ask you if you want to add any other disks before you start the actual installation. You DO because there’s the swap area.

Preparing crypto softraid

Once the installer has finished, reboot the machine. Now the boot manager says “F1 – FreeBSD” and “F2 – BSD”. The second one is your OpenBSD. The manager knows only the partition type and has no clue which system is on there.

Plain text summary

Here’s what you could type in for the shell parts of both installers:

FreeBSD


In the partitioning shell:
tcsh
dd if=/dev/zero of=/dev/ada0 bs=1m
gpart create -s mbr ada0
gpart add -a 4k -t freebsd -s 98G ada0
gpart add -a 4k -t freebsd ada0
gpart create -s bsd ada0s1
gpart bootcode -b /boot/boot0 ada0
gpart bootcode -b /boot/boot ada0s1
gpart set -a active -i 1 ada0
gpart add -t freebsd-ufs -s 2G ada0s1
gpart add -t freebsd-swap -s 4G ada0s1
gpart add -t freebsd-ufs ada0s1
glabel label clear /dev/ada0s1a
glabel label swap /dev/ada0s1b
glabel label system /dev/ada0s1d
dd if=/dev/random of=/dev/label/system bs=1m
geli init -b -s 4096 -l 256 /dev/label/system
geli attach /dev/label/system
newfs /dev/label/clear
newfs -j /dev/label/system.eli
mount /dev/label/clear /media
cp -Rp /boot /media
echo 'vfs.root.mountfrom="ufs:/dev/label/system.eli"' >> /media/boot/loader.conf
echo 'geom_eli_load="YES"' >> /media/boot/loader.conf
echo '/dev/label/system.eli / ufs rw 1 1' >> /tmp/bsdinstall_etc/fstab
echo '/dev/label/swap.eli none swap sw 0 0' >> /tmp/bsdinstall_etc/fstab
echo '/dev/label/clear /clear ufs rw 1 1' >> /tmp/bsdinstall_etc/fstab
mount /dev/label/system.eli /mnt
mkdir /mnt/clear
exit
exit

In the "final modifications" chroot:

rm -r /boot
ln -s /clear/boot /mnt/boot

OpenBSD


i
de
puffy
em0
dhcp
none
done
password
no
yes
no
no
!
dmesg | grep [ws]d0
fdisk -e sd0
setpid 1
A6
quit
disklabel -E sd0
a b
ENTER
4G
swap
a a
ENTER
ENTER
RAID
w
q
bioctl -c C -l /dev/sd0a softraid0
exit
Europe/Berlin
sd1
whole
e
Your layout here
w
q
sd0
OpenBSD
w
q
done
http
none
openbsd.cs.fau.de
pub/OpenBSD/5.9/amd64
done
done

Precomp (or: How to compress already compressed data?)

It’s a kind of strange feeling, but while half of the IT world seems to either already burn (or to tremble with fear), I can choose freely whatever topic I want to write about this month. I haven’t had a Windows box for almost a decade now and people who I work or keep in contact with, are also mostly *nix only. So this post is not about encryption or ransomware at all. It is about useful, respectable compression. Or more precise: The art of re-compressing already compressed data!

In January Precomp, a precompression utility, has been open-sourced! The first two sections tell a bit about how I became interested in this topic and in Precomp. Skip them if you don’t want to read that kind of stuff.

Compressing compressed data?

When I was young and new to PCs, I once tried to compress a ZIP archive with ACE (a lesser known archiver that once was comparable to the more popular RAR). I knew that ACE offered stronger compression and so I thought that this should make the file smaller. Just imagine my surprise when it turned out that I was wrong!

I guess that most of us have a story like that to tell, a story from our childhood when compression was nothing short of magic. Later when I begun to understand that even though it in fact does start with “m”, it’s not magic but math (a subject that I totally sucked at in school – but fortunately I grasped enough to get a rough idea on how compression works ;)). Now there was no surprise anymore: The compressed data is not well fit for any other general purpose compression method, even if it’s compressed with a weak algorithm.

How to work around that? Well, decompressing the ZIP file and creating a new ACE archive does the trick in the case mentioned above. Of course things are not always that straight forward. If they were, I wouldn’t really have much to write about right now and this post would be really, really short!

For whatever reason, compression continued to fascinate me and I loved compressing things to sizes as tiny as possible. It was fun to try out new experimental compression programs specialized on some specific types of files. I did that for years – until I had to stop due to a lack of time.

Games

Let’s fast forward some years from that failed compression experiment with ACE; I had replaced DOS 6.22 with Win95 which I had replaced with Win98 (SE) that I had replaced with WinME, … On some day I wanted to install Quake ]|[ Arena (yes, friends, I once was 1337 young enough to spell it like that!) on my main computer to get into it again for a LAN party next weekend. So I went looking for the darn CD. It took me a while but I finally found the CD case. I opened it up and… the CD itself was missing. Oh great! Since I didn’t feel like looking into all the other cases to find out into which I might have put it accidentally, I decided to just copy it off an older computer which had it already installed (ID were nice people. I don’t remember which version of Q3A it was, but there eventually was an official patch which also removed the CD check for the game so there was no need for a crack or anything).

Now, different versions of Windows didn’t always play together too well on the LAN and since my Quake installation was on a computer with an older Windows (and I didn’t have another cable at hand), I decided that I’d just burn it to CD. It turned out however, that the other machine didn’t have vanilla Q3A installed but the expansion set as well. Together it was obviously too big to fit on one CD. There would have been easy solutions: Leave out the resource files for the expansion, burn two CDs, put the hard drive into the new computer, … Sure, easy solutions are nice and all. But sometimes they are also boring! And when you’re young and have some free time, you don’t do boring stuff. So of course I opted for the more challenging solution: Get it all on one cd!

Quake 3’s resource containers go by the file extension of .pk3 and, more importantly, are in fact ZIP files without any compression. This meant that they could be compressed well because there was no ZIP compression getting in the way. But guess what: Even after applying the most extreme compression programs, the result simply would not fit onto one CD…

Bad luck, eh? Well, not really. Unpacking the container files was in fact the solution in this case. Not because of weak compression but because it enabled me to test each of the files it contained separately with all compressors and could group together all files that compressed best with one compression utility or another! I think that I was able to shrink it down almost as much as needed with just a couple of megs over the CD limit. There were blank CDs with 800 MB capacity as well, so it would have fit onto one CD – but I didn’t have one of those. So I replaced the ID video with an empty video file and I was set.

Since I liked doing these things I begun doing backups like that for a lot of my favorite games, ripping apart (and later rebuild) resource containers, convert between file formats, decompress whatever could be decompressed before applying stronger compression, etc.

How Precomp works

The more I got into free and open source things, the more I wondered if some of them wouldn’t benefit from better compression. A friend and former classmate of mine invented Precomp and I of course was among the first to make use of it and provide feedback. But what is Precomp?

Precomp is what the name says: A pre-compressor. It is not directly meant to reduce the size of files. On the contrary: It can make some files even bigger than the original input. But that’s a good thing really! How’s that? Well, it’s meant to prepare files for compression so that eventually these files can be compressed to a smaller size than the original file could – without losing data of course!

What Precomp does is look for streams in its input file that are compressed with a compression method known to Precomp. It then decompresses and recompresses them so that they can be compared. If they are identical, Precomp will write the decompressed stream (plus how to recompress it properly) to its output file.

While this sounds quite simple in theory, it is in fact a bit more complex. The reason for that lies in the flexibility of some compression algorithms. Have you ever zipped up a file? Then you know that there are a lot of parameters that you can provide which affects how the file will be compressed: “fast”, “normal”, “strong” or “maximum” compression? What about the dictionary size? A lot of things like that. So either combination of compression parameters will result in a valid zip stream that can be decompressed by any zip-compatible utility. Replacing such a stream with a compatible one is fairly easy. Reproducing the exact, bit for bit identical stream, is not.

To be truly lossless, Precomp uses trial and error on each stream. If it can figure out the combination of parameters that result in the original stream: Great! If not, that stream has to be left untouched.

What Precomp can do

Early versions of Precomp were only available on Windows but there have been Linux versions for quite a while as well. I also use it on FreeBSD without any problems. The .PCF files are platform-independent. You can restore the original file on Windows from a file precompressed on Linux or BSD and vice versa.

While Precomp originally was only a pre-compressor for zlib streams (which are used in a variety of file formats like ZIP, GZIP, PNG, PDF, …), it can do more things now. It can use bzip2 to compress its input file after precompression. It can losslessly compress some JPEG pictures to smaller sizes (thanks to an external library). And in the current development version there’s even support for compressing MP3 music files further (also using an external lib)!

Currently, Precomp relies on temporary files for all the extracted streams and thus puts heavy load on your hard drive (and is a bit slow due to that bottleneck). SSDs obviously perform better, but it totally makes sense to use a memdrive if you can spare some RAM for it. I’ve forked the project on Github and added an experimental shell script to assist with the creation of such a memdrive. It’s currently FreeBSD only (I’ve migrated all of my boxes to *BSD and currently have no Linux machine remaining but will set up one for cases like that some time in the future). Feel free to take a look at it if you’re into portable shell scripting and please do tell me if you have any suggestions!

Precomp is not at all at the limit of its possibilities. There are a lot of things that can be tweaked, optimized or added. If you feel like that could be a fun project – go ahead and play with it, it’s on Github. Or perhaps you have an idea what this could be useful for? Please help yourself and use it. It’s free software after all (Apache licensed).