Building a BSD home router (pt. 1): Hardware (PC Engines APU2)

Pretty much everybody in the western world has internet access at home these days. It’s not a big deal: You conduct a contract with some ISP. They send you a modem/router combo box that you plug in, do some simple setup and you’re done. Those boxes are pretty much ubiquitous pieces of hardware, silently doing their work most of the time. Some of them even come with wireless access and all that other convenient stuff. Basically you configure them once and then forget that you even have one!

Why would you want your own router?

The paragraph above sounds great, doesn’t it? It sure does. And it actually is. Kind of. There’s only one little problem with it, really. Those boxes are more often than not meant for sunny, green meadows where unicorns graze peacefully. Unfortunately… the internet is a less friendly place: Predators, scavengers and all kinds of poltergeists are out there and after your hide, luring in the shadows ready to rip your guts out when they catch you off guard!

Let’s put it straight: It’s not much of a secret that standard off-the-shelf type home routers suck. Big time. They do little to protect the unsuspecting user. They get in the way of people who know a bit more than the average user and try to tighten security at least a little. They have proven to often come with serious bugs. Or worse: They might even contain holes that were introduced on purpose…

If you’re running Linux, *BSD or some other OS that puts you in control, you might not be too comfortable with that potential spy in your house. (In times where people choose to put known spies like “Alexa” in their homes, your router is probably not the first thing to take care of, though. But maybe you live in a shared apartment together with someone who thinks that this devil’s work is “cool”. You know that a move is in order, don’t you? But hey, it could be worse: You could have married someone who thinks so – in which case of course you’re hosed.)

Hardware considerations

The first thing to think about – at least for me – was which software to use. There are various possibilities to choose from. Since I’ve come to appreciate BSD operating systems, I wanted something BSD based. This basic thing decided on, I needed to find hardware that was supported by my OS of choice.

However there was more to think about. While a PC is usually powered down and turned off when it’s not needed, you probably don’t want to do the same thing with your router. And while an old PC could technically do the job, for something that’s basically an “always on” device, it makes sense to use something that doesn’t draw as much power. For that reason some of those small embedded boards crossed my mind. However those are often ARM or MIPS based and sometimes don’t even have proper gigabit LAN. This doesn’t mean they are not up to the task but going with a less common architecture wouldn’t exactly make things easier. Therefore I decided that x86 was the way to go, at least for now. There’s enough new things to learn and if everything works out very well I can always play the “build your own router” game again, choosing a higher difficulty level.

Ok, x86 and (preferably) low power consumption. There are various products which fit into this category, and they often are passively cooled as well which is nice, too. I read good things about PC Engine’s APU2 boards. There are reports out there indicating that they run FreeBSD and OpenBSD really well. They also seem to be pretty popular among people building their own routers – and it’s not hard to see why.

The APU2c4

The APU2 is a system board that comes in a couple of variants. I opted for the APU2c4 which has among other features the following specs:

  • a quad-core (Jaguar) 64 bit AMD CPU
  • 4 GB DDR3 ECC RAM
  • 3 Gigabit Ethernet jacks

It makes use of Coreboot which is great and the CPU features the AES-NI instruction set that enables AES crypto acceleration which is useful as well. While the RAM is ECC technically, the firmware does not support error correction, yet, unfortunately. But that feature may be enabled with a future firmware update.

In the end I ordered a bundle that consists of the board, an external power supply, an indoor case and an mSATA 16 GB drive. The APU can make use of an SD card, too, but I definitely prefer the mSATA option. And I only paid about 180 Euros for it. Sure you can get off-the-shelf routers for a lot less, but… yeah.

Assembling my router

When I received my shipment, I opened it up and took a look at the parts. Luckily a colleague who is much more experienced with hardware than I am, offered to help me. At my first attempt I wasn’t even able to get the board into the case – it didn’t seem to fit! Of course it does fit… You just have to remove the screws for the COM port first.

APU2c4 and case

My bundle came with a small metal heat transfer plate and two stripes of double-sided sticky tape. You’re supposed to use one to use the latter to fix the plate so that it connects the plate to both the processor and the case. However my colleague asked me if I’d rather do things right and I agreed.

So he put tape on one side of the plate to stick it to the case. Then he put the board into the case. The board has two holes for screws:

Holes for screws in the board

With a marker pen he wrote marks on the on the plate to indicate where the holes on the board are. Then he removed the board again and drilled two holes through the plate and the bottom of the case. Using a screw tap he then cut two screw threads into the material.

The heat transfer plate with markers and double-sided sticky tape

The next step was to properly clean both the plate and the CPU, put thermal conductance paste on there and put the board back in place. Now the screws could be used to correctly fix the whole thing. I also plugged in the mSATA drive.

The board is fixed properly and mSATA drive plugged in

My colleague used plastic screws for the simple reason that it’s easy to cut off the overlapping parts that went through the bottom of the case. Not a bad idea!

Bottom view: Those white spots are the plastic screws

Done! I thought about painting the plastic screws’ ends black but then again that’s only the bottom of the case. I’ve been using my router for a couple of weeks now and I’m pretty happy with it (and have a lot to play with!).

UPDATE: My colleague also got a similar APU2 and was curious enough to test how much it makes a difference to use the heat plate the way the bundle suggests or to use thermal conductance paste. He put his machine under various load situations once with the sticky tape and once with the paste applied. The difference proved to be ranging between 2° to 5° C! That really makes the extra effort worth it.

What’s next?

The APU does not have any VGA port, you have to attach a serial console to work with it. So that’s what the next post will be about.

FreeBSD jails (2/2): 4.11 sentenced to jail

Ok, after introducing FreeBSD’s jails and jail frameworks in the previous post, it’s time to actually put that 4.11 system that was built in February behind bars. Before turning towards that special case, though, we’ll have a look at how to use iocage (one of the jail management frameworks).

Creating and starting an 11.0 jail

We’ve built iocage from ports last time, so let’s just jump right into it and try out one of the most important commands:

# iocage list

Iocage initialized

Not too surprisingly, the jail list is empty. But the important thing here is that iocage automatically created its infrastructure when run for the first time. This is generally fine; if you happen to have more than one ZFS pool, though, you may want to manually activate one for iocage usage.

The next thing to do is to fetch the distribution tarballs for the FreeBSD version that shall populate the jails you want to create:

# iocage fetch -r 11.0-RELEASE

Fetching the 11.0 release

This process will also update the kernel and userland to the latest available version of the selected release. I chose 11.0 because that’s the latest right now and obviously 4.11 is not available!

You can actually skip fetching the release as iocage will do it automatically if you create the jail, but doing it beforehand doesn’t hurt. Now let’s create our first jail:

# iocage create tag=fbsd11 ip4_addr="em0|192.168.2.10/24" -r 11.0-RELEASE

Jail ‘fbsd11’ created!

Each jail has a UUID. This one’s UUID begins with 99a9941a. You can interact with jails by using that UUID. But if you’ve got a lot of jails it would be pretty hard to keep track of which one is which. For that reason you can assign a tag to a jail that you can use to identify it. We’ve effectively given our jail a name: fbsd11. In general it’s a good idea to name it in a way that helps you (or your co-workers) to understand what the jail is used for. You may choose something like “postgres-jail” that includes the technology used, “loadbalancer” to describe its function or whatever fits your needs.

We’ve also assigned the jail an ipv4 address, 192.168.2.10 in this case, noted the subnet in CIDR notation and specified which NIC to use it on. Listing jails shows all that information at a glance.

The jail is ready to be started:

# iocage start fbsd11

Starting the jail

As you can see here from the ifconfig output, the ip address was not yet assigned to the NIC. According to iocage the jail has been started successfully. Let’s take a look at it:

# iocage list

The jail is running

The state for our jail has changed to “up”. Also the ip has been automatically assigned to our network interface when iocage started the jail. When the jail is stopped, it is also automatically removed again.

Using the jail

Iocage offers some means to interact jails without actually “entering” them, like e.g. installing packages, etc. The easiest way to use a jail however, is to get a shell prompt inside the jail (think entering a chroot environment). That’s what we’re going to do next:

# iocage console fbsd11

Entering the jail console

Looks pretty much like we just logged into a FreeBSD system, huh? Well, we basically did just that. With the big difference of not being a full OS (which would be the case with a VM for example) but a jail with just a FreeBSD 11.0 userland (that is running on the host machine’s kernel).

Ok, I bootstrapped pkg and then installed nginx:

Installing the nginx webserver

This also proves that the network is working since we’re receiving the packages to install from. There’s also nothing special in doing this, just use pkg the way you’re used to. For the most part it’s behaving just like a regular FreeBSD system. There are a few exceptions. One of the most notable ones is the fact that by default you won’t be able to use the ping command. You can enable jails to use ping using a sysctl but this affects the jails security and therefore is disabled by default.

Let’s start nginx next:

Nginx is running

Again it’s nothing special. Nginx is running and bound to port 80. Does it do what you’d expect it to?

Testing and destroying the jail

Nginx’s default page

Yes, the webserver is happily serving the default page when accessed from other computers on the LAN! It doesn’t make much of a difference from running nginx on the host system – unless somebody manages to use an exploit. Nginx is privilege separated and thus probably not a good example. But think about a case where an intruder manages to get into your system and get root privileges. That’s pretty bad, yes. But if the application was running in a jail, the damage is at least contained and the other jails as well as the host system are unaffected. (That and it’s much easier to tear that jail down and create a new one compared to having to reinstall the whole host system and setup several applications again!)

Alright. Let’s take a quick look at where the jail data is kept:

Taking a look at iocage’s directory structure

Iocage created the /iocage dataset. The most important path is /iocage/jail. All the jails are in there (named after their UUID). To make accessing them easier, there’s /iocage/tags: Here you’ll find symlinks named after the jail’s tags which are pointing to the actual directories with the data.

Each jail consists of a configuration file, an fstab and a “root” directory (which holds all the files).

This jail has served its purpose and is no longer needed. So let’s get rid of it! First exit the jail like you would exit a chroot (e.g. CTRL-D out of it) before you proceed:

# iocage stop fbsd11
# iocage destroy fbsd11

Stopping and destroying the jail

You cannot destroy a running jail. Therefor it must be stopped first and can then be destroyed.

Creating the 4.11 jail

That has been pretty easy and straight forward, right? And here’s the good news: Jailing the 4.11 system isn’t that difficult, either. I created a tarball of the complete filesystem on the FreeBSD 4.11 server, xz compressed it and scp’d that onto the jailhost. Now we’ll create another 11.0 jail, delete the filesystem and extract the old OS data into the same place:

# iocage create tag=fbsd4.11 ip4_addr="em0|192.168.2.10/24" -r 11.0-RELEASE
# cd /iocage/tags/fbsd4.11/root
# rm -rf * .*
# tar xvJf /path/to/tarball.xz

Creating a new jail and putting the 4.11 OS into it

The old device nodes won’t work on a current kernel and have to be removed. The same goes for the old proc filesystem. We also don’t need any kernels or kernel modules. And /boot is obsolete in a jail, too. So it’s a good idea to get rid of all that:

# rm dev/*
# rm kernel*
# rm -rf boot
# rm -rf modules*
# rm -rf proc/*

Deleting unneeded parts of 4.11

The old userland and the additional data remains. Let’s try to start the jail now:

# iocage stop fbsd11

Starting the 4.11 jail

Uhh… This doesn’t look good. Iocage tries to do things that won’t work with a 4.11 userland (anything older than 9.3 is unsupported with iocage, anyways!). But fortunately that looks worse than it actually is:

It errors out… but it’s running!

Despite those errors the jail is up and running! Can we get console access?

Inside the 4.11 jail

# iocage console fbsd4.11

Entering the console works, too!

Yes, we can! Let’s make 4.11 great again! Time to install a webserver in this jail, too. Since we still have Pkgsrc in place, that’s an easy thing to do:

# cd /usr/pkgsrc/16/www/nginx
# bmake install clean clean-depends

Building nginx on 4.11 through Pkgsrc

I decided to modify nginx’s default starting page so it looks a little different from the stock one. Nginx can be started simply by executing the daemon. However all the system utilities regarding process management don’t work! And sockstat is not working either.

Starting nginx, but tools like ps don’t work!

Does nginx actually run? There’s an easy way to test that. Let’s just open up a browser on another pc and see if we can get the modified webpage.

Nginx is able to serve the webpage from the 4.11 jail!

It works! Yes, it was actually that easy. Jails are not necessarily a complicated thing at all (they can be more complicated than that of course, if you’re doing more advanced stuff). But… No ps, no top, no kill… How are you supposed to work with a 4.11 jail?

Peter Wemm suggested using statically compiled binaries from the host system like those in /rescue. And that’s actually a great idea! If you copy /rescue over into the jail, it at least provides some tools to work with.

Thanks to /rescue process management is possible

Things like top would be nice, too, of course. But to be honest, I don’t know how to statically build them. If anybody knows, please comment and share your wisdom!

More about jails?

This article merely gave a very basic introduction to the topic (plus a little “stunt”). You can do a lot more with jails. I might write about it again in the future but I haven’t decided, yet (feel free to comment and tell me what you’re interested in!). But I wanted to at least give you an idea what other things there are to explore.

If you have a server with only one ip you can still use jails. The solution to this is to give the jails internal network addresses (like 192.168.x.x) and then use Pf to do network address translation (NAT). It took me a bit to figure out how this works (there are things on the net that are better documented) but in the end it’s not that complicated.

With VIMAGE/VNET you can give your jails their own virtual network stack. That way you can even use a firewall inside the jail and assign multiple ip addresses!

Jails support quotas of various kinds. You can limit the amount of CPU that one jail may occupy, limit file system IO, etc. Lots of possibilities with this.

And of course iocage has many nice features. You can e.g. create jail templates with some software already installed and quickly create jails from those. There are multiple jail types like e.g. thin-provisioned jails. You can clone and snapshot jail. And you can have jails started automatically when your server boots.

In the previous article I said “think containers” when you’re completely new to jails. Now you’ve seen a bit of what they can do and have a better idea of what they are. So stop “thinking containers” now. These are two things that are somewhat related but they are not the same thing. Linux’s containers can do things that jails were never designed for (e.g. sharing namespaces). But jails have one huge advantage over containers: While containers are only a means for logical separation, jails were designed for security right from the start. Sure, people are working on making Linux’s containers more secure. But if there’s one thing that we should have learned by now, it is this: Retrofitting security into something is not really going to work… And jails are pretty flexible, too. You might not have missed them when you didn’t know that they exist. But get used to using them once and you’ll think differently about a lot of things and make use of them for more and more things. Jails are a great feature of FreeBSD.

FreeBSD jails (1/2): Introduction and frameworks

This is the first part of a followup to the 4.11 mini-series of posts that I published here on my blog ([1], [2], [3] & [4]). Those posts covered installing an old FreeBSD 4.11 system and using pkgsrc to update some parts of the system from versions released in the last millennium (!) to those of today (or rather from December 2016).

Now we’re going one step further: We’ll migrate a 4.11 system into a jail on a fresh FreeBSD 11.0 host system! This will be dealt with in the next post. In this part I’ll introduce the jail concept for people who are not familiar with it, discuss some jail management frameworks and do a bit of preparation work.

Motivation

Why to send that old system to a jail? We’ve already done a bit of work on a 4.11 system. But that next step tightens security much more since we have a recent kernel, can make use of a state-of-the-art firewall, etc. This is not actually my main concern however. Those remaining machines are running internal processes, so security is nice but not critical. What is critical and really frightens me, is the age of the hard drives that they use… When you do a df and see that the total capacity of a drive is way below 100 GB, you can guess that drives like that have not been sold for… quite some years now! And knowing that drives are rather perfidious little things, I’d rather get rid of those as soon as possible.

What makes things worse: This is 4.11. There’s no GEOM available and thus no gmirror or anything. Yes, there are backups. But I’d rather not have to mess with something like that just to restore a system that should have been decommissioned years ago!

What are FreeBSD jails?

If you’re new to FreeBSD, you may have heard about jails without knowing what they actually are. If you’ve got a Solaris background, think zones. If you’re a Linux user, think containers. FreeBSD people usually cannot understand today’s container hype for the very simple reason that this OS has had jails for ages now (they were first available on development versions at the very end of the 20th century!). You are probably familiar with docker, Linux’s cgroups and all those facilities. FreeBSD basically does the same thing: Provide much enhanced but still very light-weight chroot environment.

The system in that jail is isolated from the actual host operating system so that escaping it isn’t trivial and requires special (insecure) settings. In general jails are meant to be secure after all, aren’t they? You can also give jails their own virtual network stack, you can cap their resource usage, and so on. It’s a mature, stable and secure way to do containerization of applications – or whole operating system installations.

Since FreeBSD is pretty good at keeping compatibility with old releases (for the next major release they are discussing whether to keep compatibility shims that allow executing binaries from commercial UNIX releases of the 80’s!!), it’s very much possible to stuff a 4.11 userland into a jail running on a recent version of the operating system. And that’s what we’re going to look at after getting some of the basics out of the way.

Jail frameworks

Jails in FreeBSD can be created and managed manually. We’re talking *nix here, after all! However there are several jail management frameworks available that promise to make things even easier or at least more convenient. If you want to get started with jails, I suggest that you pick a framework that sounds good to you feature-wise and simply begin playing with it. The whole thing is not that complicated actually.

If you’re just getting started, don’t bother with sysutils/warden. It was a popular jail manager created for PC-BSD but it is considered obsolete now.

One true classic among the jail frameworks is sysutils/ezjail. It still does the job and a lot of people continue to use it simply because they are already familiar with it. It allows for things like thin-provisioned jails that use a “base jail” which holds a complete FreeBSD userland, and uses nullmount to make that accessible in other jails. This makes updating very quick and convenient. It also leaves the base system read-only which may be another nice security feature at the cost of decreased flexibility. On the plus side Ezjail is independent on filesystems and thus works on UFS, too.

Then there was sysutils/iocage, a modern approach that made use of ZFS properties to store certain values. For that reason it demands at least one zpool on the system to be usable. It had been implemented in shell code which made it easy to read for admins who don’t have a lot of coding experience. It has however been rewritten and the old shell version is declared obsolete. If you like the idea of having it written in shell, take a look at sysutils/iocell, which is a fork of the last shell version and is actively maintained.

The new version of iocage is implemented in Python (initially it was announced to be rewritten in Go but that decision was revoked). Another warning: sysutils/py-iocage is the wrong version, too! Do not use it. The rewrite initially supported Python 2.7 – which is what this port is all about – but newer versions dropped support for that and now require Python 3.6. So the actual port you should use is sysutils/py3-iocage! That version requirement makes sense but unfortunately it leads to iocage not being available from packages (as FreeBSD’s default version of Python 3 is still 3.3 currently). So you probably need to build it from ports.

There’s one more framework that I want to mention: sysutils/cbsd. It is an advanced system that not only does jails but also allows to manage bhyve (FreeBSD’s modern hypervisor). If you’re looking for something more comprehensive you may well give it a try.

Choosing my framework

When I first wanted to get into jails, I tried out ezjail. It worked well for me and I’d still use it on a system with no ZFS available. When I have ZFS, though, I liked the iocage approach. Cbsd looks quite good but I didn’t do too much with it because I require things to be scriptable (like using the framework through SaltStack states – there’s even a Salt Formula for iocage, but that’s for the old implementation) and that doesn’t seem to be what they focus on. So when the shell based iocage was deprecated, I moved to iocell and then I was torn to stick with that or to return to the new iocage. In the end I opted for at least giving the rewrite a chance. Since it does its thing quite nicely, I’ve adopted that and will use it here.

You can of course use any other framework if you prefer a different one. Or you could setup your jail by hand – if you’re thinking about that, definitely take a look at this article by M. W. Lucas whom I actually have to thank not only for his post but kind of for this one as well. The thought of jailing the old systems crossed my mind once or twice, but if he hadn’t written about it in the past, I don’t think I’d ever have tried it (no, I read his post back in the day when he published it; yes, I wanted to try this for over three years now before I finally found some time to give it a shot!).

Preparations

Let’s install iocage first. Like stated before, we will have to build it from ports. Assuming a system that does not have a ports tree installed, this is how you can do it (there are better (cleaner) ways to build packages from ports, I planned to do a post about that topic since December!) but this old method still works:

# portsnap fetch extract
# cd /usr/ports/sysutils/py3-iocage
# make config-recursive
# make install clean

The new iocage needs a UTF-8 locale set on the system or it won’t run. Unfortunately this is not the standard in FreeBSD (this is something that I really hope for in FreeBSD 12!). There are multiple ways to do this, my preferred one to set it in /etc/login.conf:

Look for default:\ – it is followed by a block of indented lines, the last one being :umask=022:. Add another backslash at the end to make the block continue and add two more lines to it:

:charset=UTF-8:\
:lang=en_US.UTF-8:

Login configuration is one of the few things where the actual values are in fact stored in a database. The changes that you made to the text file won’t take effect even if you log out and back in. First run another command to update the DB according to the changes in the text file:

cap_mkdb /etc/login.conf

Now log out and back in. Try echoing $LANG – that should print the UTF locale’s name. If it does, you are set.

What’s next?

The next post will demonstrate how to use iocage to manage jails and show how to jail a FreeBSD 4.11 system so it runs on a FreeBSD 11 host.

Updating FreeBSD 4.11 (4/4) – Reflecting radical resurrection

In the first post of this mini series I wrote about legacy systems and installing FreeBSD 4.11. The second one shows how to configure the fresh system for remote access, bootstrap Pkgsrc, install Subversion to checkout FreeBSD code and update the system to the stable branch. And part three mainly deals with upgrading OpenSSH and the compilers. This post details some more updates until we reach the final state that’s possible with such an old system (without resorting to extreme means).

Planting a new tree

So far we’ve built some packages from 2013 and before. Using a current pkgsrc tree won’t work – the various pkgsrc tools that our system has are too old. It might not be too big a step but we can use a tree from the second half of 2014. Of course the newer SSH that we built before is not currently in the path so we need to create a temporary symlink before we can use CVS again:

# ln -s /usr/local/temp/bin/ssh /usr/local/pkgsrc/bin/ssh
# rehash

# cd /usr/pkgsrc
# cvs -danoncvs@anoncvs.netbsd.org:/cvsroot get -rpkgsrc-2014Q3 -P pkgsrc
# mv pkgsrc 14
# rm /usr/local/pkgsrc/bin/ssh

What the system looks like package-wise at the beginning of part 4

Most of pkgsrc’s tools make use of NetBSD’s compatibility library. Unfortunately the version that comes with the new pkgsrc tree won’t build anymore on an OS as old as FreeBSD 4.11. Same thing for libfetch. But the newer tools will work with older versions of that libs, too. So let’s prepare those two – libfetch need’s some more love to build:

# cd /usr/pkgsrc/13/pkgtools/libnbcompat
# bmake

# cd /usr/pkgsrc/13/net/libfetch
# cp Makefile Makefile.bak
# sed '14i\\
CFLAGS=         -Wno-error' Makefile.bak > Makefile
# bmake

As a next step we’re going to do two updates. Yes, in theory we could use “bmake update” to update packages. We will not do that. The reason is that we needed to abuse pkgsrc quite a bit so far by mixing package versions from various trees. Since “bmake update” is a destructive command (it will happily uninstall programs as well as packages depending on them!) this can lead to all sort of fun things like unresolvable dependencies and such.

If you like pain, go ahead. I’ve been there and I can confirm that it does work for some packages. For a lot of them actually. But in those cases where it doesn’t, it tends to do so much damage that you’re better off starting over than trying to fix things… That’s why I’ll show you a safer method instead: Build a package and update via pkg_add! Also it really starts to show how old the system is that we’re trying to build rather new packages on. More and more of them require some trickery to persuade them to build – but hey, we’re doing a gross thing here, anyway. So there’s no real reason to complain!

# cd /usr/pkgsrc/14/pkgtools/pkg_install
# bmake extract
# rm -r work/libnbcompat/*
# rm -r work/libfetch/*
# cp -R /usr/pkgsrc/13/pkgtools/libnbcompat/work/libnbcompat-20120702/* /usr/pkgsrc/14/pkgtools/pkg_install/work/libnbcompat/
# cp -R /usr/pkgsrc/13/net/libfetch/work/libfetch-2.34/* /usr/pkgsrc/14/pkgtools/pkg_install/work/libfetch/
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/14/packages/All/pkg_install-20130902nb1.tgz

# cd /usr/pkgsrc/14/pkgtools/bootstrap-mk-files
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/14/packages/All/bootstrap-mk-files-20140516.tgz

We made it so far, now let’s make a daring move and just download the latest stable pkgsrc tree – released in January 2017:

# cd /usr/pkgsrc
# fetch http://cdn.netbsd.org/pub/pkgsrc/stable/pkgsrc-2016Q4.tar.bz2
# tar xvjf pkgsrc-2016Q4.tar.bz2
# rm pkgsrc-2016Q4.tar.bz2
# mv pkgsrc 16

Updating pkgsrc tools

Since mid 2014, pkgsrc makes use of a new package, cwrappers. During my test run I somehow managed to just get this package built. Despite taking notes I have no idea what I did to just make it work! It must have been something that looked like a dead end (which is why I didn’t include it in my notes) but somehow provided “getline”… I tried to get it working again for almost one whole Sunday but for the life of me couldn’t find out what I previously did… In the end I gave up and tried to find another solution. I found one but while it is way more complex it at least means that I got rid of that nasty blocker again:

# cd /usr/pkgsrc/16/pkgtools/cwrappers
# bmake extract
# rm -r work/libnbcompat/*
# cp -R /usr/pkgsrc/13/pkgtools/libnbcompat/work/libnbcompat-20120702/* /usr/pkgsrc/16/pkgtools/cwrappers/work/libnbcompat/
# cp work/cwrappers-20161125/mi_vector_hash.c work/cwrappers-20161125/mi_vector_hash.c.bak
# cp work/cwrappers-20161125/fixup-libtool.c work/cwrappers-20161125/fixup-libtool.c.bak
# sed 's/stdint.h/inttypes.h/' work/cwrappers-20161125/mi_vector_hash.c.bak > work/cwrappers-20161125/mi_vector_hash.c
# sed 's/stdint.h/inttypes.h/' work/cwrappers-20161125/fixup-libtool.c.bak > work/cwrappers-20161125/fixup-libtool.c
# cp /usr/pkgsrc/14/pkgtools/cwrappers/files/bin/getline.c /usr/pkgsrc/16/pkgtools/cwrappers/work/cwrappers-20161125/getline.c.bak
# sed 's/ssize_t/size_t/' work/cwrappers-20161125/getline.c.bak > work/cwrappers-20161125/getline.c
# cp work/cwrappers-20161125/common.h work/cwrappers-20161125/common.h.bak
# sed '107i\\
size_t  getline(char **, size_t *, FILE *);' work/cwrappers-20161125/common.h.bak > work/cwrappers-20161125/common.h
# cp work/cwrappers-20161125/Makefile work/cwrappers-20161125/Makefile.bak
# sed '14i\\
LIB_SRCS+=      getline.c' work/cwrappers-20161125/Makefile.bak > work/cwrappers-20161125/Makefile
# bmake install clean clean-depends

Phew! Fortunately the next few updates are straight forward:

# cd /usr/pkgsrc/16/pkgtools/bootstrap-mk-files
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/bootstrap-mk-files-20160908.tgz

# cd /usr/pkgsrc/16/devel/bmake
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/bmake-20150505.tgz

# cd /usr/pkgsrc/16/net/tnftp
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/tnftp-20151004nb1.tgz

Next is another one that requires some patching:

# cd /usr/pkgsrc/16/pkgtools/digest/
# bmake extract
# cp work/digest-20160304/sha3.h work/digest-20160304/sha3.h.bak
# cp work/digest-20160304/keccak.c work/digest-20160304/keccak.c.bak
# cp work/digest-20160304/keccak.h work/digest-20160304/keccak.h.bak
# cp work/digest-20160304/sha3.c work/digest-20160304/sha3.c.bak
# sed 's/stdint.h/inttypes.h/' work/digest-20160304/sha3.h.bak > work/digest-20160304/sha3.h
# sed 's/stdint.h/inttypes.h/' work/digest-20160304/keccak.c.bak > work/digest-20160304/keccak.c
# sed 's/stdint.h/inttypes.h/' work/digest-20160304/keccak.h.bak > work/digest-20160304/keccak.h
# sed 's/stdint.h/inttypes.h/' work/digest-20160304/sha3.c.bak > work/digest-20160304/sha3.c
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/digest-20160304.tgz

Updating installed packages

Let’s update gettext first as a lot of packages need that one; xz is one of the packages that is linked against the old one and since libintl received a soname bump, it needs to be rebuilt. Since we want to update it anyway that’s not too bad. But there are other packages that we cannot update which depend on the old lib. So we’ll have to create a symlink to satisfy their need, too:

# cd /usr/pkgsrc/16/devel/gettext-lib
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gettext-lib-0.19.8.1.tgz

# cd /usr/pkgsrc/16/archivers/xz
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/xz-5.2.2.tgz

# ln -s /usr/local/pkgsrc/lib/libintl.so.9 /usr/local/pkgsrc/lib/libintl.so.7
# cd /usr/pkgsrc/16/devel/gettext-tools
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gettext-tools-0.19.8.1.tgz

Next in line is some more typical build dependencies:

# cd /usr/pkgsrc/16/devel/libtool-base
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/libtool-base-2.4.2nb13.tgz

# cd /usr/pkgsrc/16/devel/m4
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/m4-1.4.17.tgz

# cd /usr/pkgsrc/16/devel/bison
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/bison-3.0.4nb3.tgz

Just a few more packages and we’ll have updated most packages that can be updated (a few like zip and nbpatch can’t):

# cd /usr/pkgsrc/16/shells/bash
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/bash-4.4.005.tgz

# cd /usr/pkgsrc/16/lang/perl5
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/perl-5.24.0.tgz

# cd /usr/pkgsrc/16/devel/autoconf
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/autoconf-2.69nb7.tgz

# cd /usr/pkgsrc/16/devel/gmake
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gmake-4.1nb3.tgz

Rebuilding the compiler

First we need to update the two math libraries (and create another symlink so we can go on compiling):

# cd /usr/pkgsrc/16/devel/gmp
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gmp-6.1.2.tgz
# ln -s /usr/local/pkgsrc/lib/libgmp.so.13 /usr/local/pkgsrc/lib/libgmp.so.11

# cd /usr/pkgsrc/16/math/mpfr
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/mpfr-3.1.5.tgz

This unfortunately breaks the compiler. But we can still resort to the old GCC3 to build GCC4 again, right? Right:

# cp /usr/pkgsrc/13/distfiles/gcc-4.4.7.tar.bz2 /usr/pkgsrc/16/distfiles
# cp /root/.cshrc /root/.cshrc.bak
# sed 's:pkgsrc/gcc44:temp/gcc34:' /root/.cshrc.bak > /root/.cshrc
# source /root/.cshrc
# cc -v

While we probably still don’t need Object-C or Java we could in fact build GCC with them this time. Java requires Python2.7 installed but that can actually be built from the 2014 tree! The problem is that building Java requires more RAM than is available on 32 bit machines and will for that reason fail. However Java is deactivated by default for GCC 4.4 in the 2016 tree. So let’s just get rid of our custom options, build the default package and set the correct path again:

# cp /usr/local/pkgsrc/etc/mk.conf /usr/local/pkgsrc/etc/mk.conf.bak
# sed '/PKG_OPTIONS.gcc44/d' /usr/local/pkgsrc/etc/mk.conf.bak > /usr/local/pkgsrc/etc/mk.conf
# cd /usr/pkgsrc/16/lang/gcc44
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gcc44-4.4.7nb7.tgz

Almost everything updated!

Now we only have to restore the correct path and then we have the GCC4 back (with a newer patch level):

# cp /root/.cshrc.bak /root/.cshrc
# source /root/.cshrc
# cc -v
gcc version 4.4.7 (GCC)

Modern OpenSSH

There’s one more package to build that needs a bit of care: Pkgconf. It’s a simpler replacement for the older pkg-config but it won’t work out of the box for us:

# cd /usr/pkgsrc/16/devel/pkgconf
# bmake extract
# cp work/pkgconf-1.0.1/libpkgconf/stdinc.h work/pkgconf-1.0.1/libpkgconf/stdinc.h.bak
# cp work/pkgconf-1.0.1/getopt_long.h work/pkgconf-1.0.1/getopt_long.h.bak
# sed 's/stdint.h/inttypes.h/' work/pkgconf-1.0.1/libpkgconf/stdinc.h.bak > work/pkgconf-1.0.1/libpkgconf/stdinc.h
# sed 's/stdint.h/inttypes.h/' work/pkgconf-1.0.1/getopt_long.h.bak > work/pkgconf-1.0.1/getopt_long.h
# bmake install clean clean-depends

Finally the time has come to do what I wanted to do in the first place, provide a recent version of OpenSSH! Of course it’s also necessary to generate new host keys once more. And then, just to prove everything works when the machine boots, let’s just restart the machine after adjusting the sshd path:

# cd /usr/pkgsrc/16/security/openssh
# bmake install clean clean-depends
# rehash
# ssh -V

# ssh-keygen -f /usr/local/pkgsrc/etc/ssh/ssh_host_rsa_key -N '' -t rsa
# ssh-keygen -f /usr/local/pkgsrc/etc/ssh/ssh_host_dsa_key -N '' -t dsa
# mkdir -p /usr/local/pkgsrc/run

# cp /etc/rc.conf /etc/rc.conf.bak
# sed 's:temp/sbin:pkgsrc/sbin:' /etc/rc.conf.bak > /etc/rc.conf
# shutdown -r now

Generating new host keys for OpenSSH

Now we need to remove the vierelf entry in ~/.ssh/known_hosts before we connect again. Doing so in verbose mode even shows that the 4.11 box now has a newer version of OpenSSH installed that the FreeBSD 11 workstation that I use to connect to it! 😀

FreeBSD 4.11 running a newer OpenSSH than my FreeBSD 11.0 workstation!

Conclusion

FreeBSD 4.11 is really, really, really old now. But you can get surprisingly far in running somewhat modern software on it – more recent software at least than I initially thought would be possible! And you? What was your bet? Would you have guessed that I’d make it up to the 2016Q4 pkgsrc tree and even install the latest version of OpenSSL and OpenSSH?

Here’s a little summary of some important program updates:

binutils 2.12.1 (2002) -> binutils 2.17 (2006)
perl 5.005 (1998) -> perl 5.8 (2002) -> perl 5.18 (2013) -> perl 5.24 (2016)
GCC 2.95.4 (2001) -> GCC 3.4.6 (2006) -> GCC 4.4.7 (2012)
OpenSSH 3.5 (2002) -> OpenSSH 4.6 (2007) -> OpenSSH 7.3 (2016)

Not too bad, eh? The notable exception here is binutils. Newer versions would probably be possible but there’s a gap in pkgsrc – which stuck with 2.17 for a long time and then directly moved to 2.22 which no longer builds on FreeBSD 4.11. GCC 4.5.3 does build BTW but something goes sideways and the comparison of stage 2 and 3 fails for quite some files.

I’ve met my initial goal to provide a newer version of OpenSSH, surpassing all expectations that I had. There’s room for more of course but that’s not worth another post. I’m going to add sudo and since Python 2.7 can be built it might even be possible to manage the 4.11 servers using salt-ssh (the ordinary SaltStack doesn’t work as it requires ZeroMQ which looks like it cannot be built)! We have a recent version of bash and can thus do some pretty nifty things with the right .bashrc.

This whole adventure took far longer than I had anticipated – a bit over a month instead of the intended two weekends! But that was mostly because I decided to start over with a clean system several times to ensure that everything works as I wrote it down here (and because GCC4 simply takes so long to build on the only spare machine that I had for this…). But it has been an interesting ride and I don’t regret spending some time on the legendary FreeBSD 4.11!

Oh, and my special thanks to everybody involved with Pkgsrc! I usually don’t have much use for NetBSD but Pkgsrc is extremely useful. I might use it in the future on other systems (like Linux), too. And thanks to you for reading. I hope that you enjoyed it as well!

Updating FreeBSD 4.11 (3/4) – Neophyte’s notorious necromancy

The first post of this mini series was about legacy systems in general and about what installing the old FreeBSD 4.11 is like. In the second one I showed the initial configuration of the system, how to SSH into it despite the obsolete DSA host key and how to bootstrap pkgsrc, NetBSD’s portable ports tree. I also covered the installation of SVN, checking out of the 4.11-STABELE code and updating the system. This post will cover installing newer software.

Any bets?

So far we have a pkgsrc tree from mid 2007 and things seem to be working. However that’s pretty close to 4.11’s release in 2005 and thus not too amazing. Working with such an old system there are plenty of cases which mean “game over”. Here are just three errors of that kind which you can encounter trying to build more modern software:

/usr/libexec/elf/ld: cannot find -lpthread

There’s no modern pthreads available on 4.11. Game over.

/usr/include/sys/resource.h:58: error: field 'ru_utime' has incomplete type

We’ll have to do with very old system headers missing a lot of what we take for granted today. Game over again.

fileio.o(.text+0x354): undefined reference to `towupper'
collect2: ld returned 1 exit status

Sorry, that ancient libc that we have on our system doesn’t provide that symbol. Game over yet again.

How far do you think can we take it in building and installing more recent software? Make a guess now and see if you were right! To be honest I was not expecting the end result. Not at all. So let’s get back to work!

In for a screening

We’re going to compile a lot of stuff this time – building SVN and dependencies was just a warm up. And what do you do when you’re building stuff remotely over SSH? You’re doing so in a screen or tmux session of course. Neither is part of the base system so we’ve got to build one. Tmux was not yet available in 2007 so it’s not too hard a choice:

# cd /usr/pkgsrc/07/misc/screen
# bmake install clean clean-depends
# rehash
# screen

GNU screen started up and ready

If you don’t know screen do some reading because you will want to start using it (or rather the superior tmux). It basically allows you to detach from a session and reconnect later – and your programs will continue running on the remote system even while you’re logged out. You can also resume the session from another terminal or computer, share sessions, etc. And that’s just one of the things that it does. There are other features like allowing you to have multiple shell instances in just one terminal between which you can switch back and forth (think tabs of a browser) and a lot more. Should you not like this (what’s wrong with you?!), fine. Don’t install screen. It’s optional.

Replacing the front door lock

Now it’s time to take care of the main problem of our system: That darned version 3.5 of OpenSSH! Let’s build whatever our pkgsrc tree has to offer:

# cd /usr/pkgsrc/07/security/openssh
# bmake install clean clean-depends
# rehash
# ssh -V
OpenSSH_4.6p1, OpenSSL 0.9.7d-p1 17 Mar 2004

Still far from a modern version of OpenSSH but also a lot better. And the best thing: It supports RSA keys. Let’s generate host keys with this newer SSH and make it the version that FreeBSD launches during startup:

# ssh-keygen -f /usr/local/temp/etc/ssh/ssh_host_key -N '' -t rsa1
# ssh-keygen -f /usr/local/temp/etc/ssh/ssh_host_rsa_key -N '' -t rsa
# ssh-keygen -f /usr/local/temp/etc/ssh/ssh_host_dsa_key -N '' -t dsa
# mkdir -p /usr/local/temp/run
# echo 'sshd_program="/usr/local/temp/sbin/sshd"' >> /etc/rc.conf

Ok, everything is in place. We could reboot now – or just kill off the old daemon and launch the new one. Let’s first look for SSHD and see which PID it has (this of course varies from system to system!):

# ps aux | grep sshd

Replacing SSHD

Got it? Great, let’s kill it (your SSH connection is maintained by a child and it’s generally save to kill the parent. You won’t lose your SSH connection!), start the new one and ensure that it’s running:

# kill [PID on your system]
# /usr/local/temp/sbin/sshd
# ps aux | grep sshd

What’s this? It looks like it’s not running! Yes, it looks like it but actually it should be running… Let’s grep again:

# ps aux | grep local

This does return one process – and trust me it’s actually our new sshd. What’s happening here is this: The output of ps is truncated because more wouldn’t fit on the screen. And only that data is handed to grep! So the process with the name /usr/local/temp that we found (see the screenshot above) is actually /usr/local/temp/sbin/sshd with the last part of it cut off… This is why grep doesn’t find “sshd”. There’s a funny way to fix this, though: Maximize your terminal emulator so that more space is available. Then grep will find sshd!

Now we can quite the old SSH session so we can make one with the new server. We can even keep our screen session open, but we need to detach from it by pressing CTRL-A and then D before we logout from vierelf:

[detached]

# logout
> exit
Connection to 192.168.1.5 closed.

Time to edit your known hosts and get rid of the former host key for vierelf or else you’ll see that scary SSH warning when you try to login again. Oh, and you can leave out that compatibility option from now on – which is a major step ahead! When you’re back in, you can resume the screen session:

% ssh kraileth@192.168.1.5
> su -
# screen -r

Connecting to the new SSH server (debug mode)

Compiler: from antediluvian to ancient

Alright. Currently we have the last version of the second generation of GCC on our system. We totally need to get our hands on something newer. How about updating the last version of generation three? Let’s try that! We only want the C and C++ compilers. Fortran is deactivated by default for this version (it would need GMP installed and the version of GMP that’s in the tree requires GCC3. It’s a good idea to avoid that potential circular dependency). However Java and Object-C are activated. There’s no need to waste time on them, they should be deactivated as well. The following sed command may look a bit complex, but it’s not that bad. Just copy all three lines that make up that single command and you’re good to go:

# cd /usr/pkgsrc/07/lang/gcc34
# cp Makefile Makefile.bak
# sed -e '64,65d' -e '63a\\
BUILD_JAVA?=    NO' -e '63a\\
BUILD_OBJC?=    NO' Makefile.bak > Makefile
# bmake install clean clean-depends

After installing that newer GCC, the path needs to be changed again so that the system picks it up instead of the older system compiler:

# vi /root/.cshrc

Prepend the following path to the PATH variable:

/usr/local/temp/gcc34/bin

Now let’s log out and in again and see if the new compiler is available:

# exit
[screen is terminating]
# logout
> su -
# screen
# cc -v
[...]
gcc version 3.4.6

Updating pkgsrc

Since we also have a more recent OpenSSH now, we can checkout a newer copy of pkgsrc from CVS! That takes a while, be patient. Even after it is finished downloading (and you see no new lines on the screen) it will still take some time to clean things up. This is normal and you have to wait a little longer. Don’t CTRL+C it as that would leave your tree in bad shape!

# cd /usr/pkgsrc
# cvs -danoncvs@anoncvs.netbsd.org:/cvsroot get -rpkgsrc-2009Q4 -P pkgsrc
# mv pkgsrc 09

Thanks to the newer SSH: CVS works now, too!

We’ll need some ports from there later. But since we have GCC 3 available now we can also grab an even newer copy and primarily use that one:

# cvs -danoncvs@anoncvs.netbsd.org:/cvsroot get -rpkgsrc-2013Q2 -P pkgsrc
# mv pkgsrc 13

We’re going to start a fresh environment, using only GCC (and sshd) from the old one. To do so we first bootstrap the pkgsrc from 2013 into a new directory:

# mkdir /usr/local/pkgsrc
# cd /usr/pkgsrc/13/bootstrap
# ./bootstrap --prefix=/usr/local/pkgsrc --varbase=/usr/local/pkgsrc

The next step is to adjust the path variable so that the binaries from the new location are being used. To do so we need to replace /usr/local/temp with /usr/local/pkgsrc for both sbin and bin. Don’t change the compiler path, though! GCC 3 will remain in temp. After logging out and back in, screen is no longer in PATH so we need to execute it with the absolute path:

# cp /root/.cshrc /root/.cshrc.bak
# sed -e 's:temp/bin:pkgsrc/bin:' -e 's:temp/sbin:pkgsrc/sbin:' /root/.cshrc.bak > /root/.cshrc
# exit
# logout
> su -
# /usr/local/temp/bin/screen

Cherry-picking dependencies

This gives us a way to easily build software from 2013. Let’s continue on by fetching some source tarballs by hand that are no longer available on the mirrors that pkgsrc knew for them:

# cd /usr/pkgsrc/09/distfiles
# fetch http://ftp.cc.uoc.gr/mirrors/NetBSD/packages/distfiles/binutils-2.17.tar.gz
# fetch http://ftp.cc.uoc.gr/mirrors/NetBSD/packages/distfiles/pkg-config-0.23.tar.gz

The following part is not too interesting: We’re going to build the dependencies in preparation for the next big step. In general we try to build the newest version possible (2013) but resort to old (2009) or even older (2007) where necessary if newer versions don’t build for various reasons:

# cd /usr/pkgsrc/13/converters/libiconv
# bmake install clean clean-depends

Zip from 2009 and onwards is incompatible with FreeBSD 4.11’s libc. And the 2007 version expects tar in a location where there’s none on our system. Instead of building tar we can safely symlink it:

# ln -s /usr/bin/tar /usr/local/pkgsrc/bin/tar
# cd /usr/pkgsrc/07/archivers/zip
# bmake install clean clean-depends

The binutils are a special case. The port normally builds the programs of which it consists with a prefix so they don’t get in the way of the system binaries. Since we actually want to use them instead of the old stuff from the base system, we need to get rid of that prefix:

# cd /usr/pkgsrc/09/devel/binutils
# bmake GNU_PROGRAM_PREFIX='' install clean clean-depends
# rehash
# ld -v
GNU ld version 2.17

The next few are trivial:

# cd /usr/pkgsrc/09/devel/gettext-tools
# bmake install clean clean-depends

# cd /usr/pkgsrc/13/devel/m4
# bmake install clean clean-depends

# cd /usr/pkgsrc/09/devel/bison
# bmake install clean clean-depends

The bash port from 2013 would draw in a newer version of gettext which would not build. But bash can actually be built with the old one, too. So we have to make a simple change in the buildlink file for gettext in 2013’s pkgsrc tree:

# cd /usr/pkgsrc/13/devel/gettext-lib
# cp buildlink3.mk buildlink3.mk.bak
# sed 's/0.18/0.14/g' buildlink3.mk.bak > buildlink3.mk

With that change the next port can be built:

# cd /usr/pkgsrc/13/shells/bash
# bmake install clean clean-depends

Next in line is perl. The 2013 port would however build with dtrace support by default – which was of course not available on 4.11. Therefore it needs to be switched off by making an addition to the pkgsrc config file:

# vi /usr/local/pkgsrc/etc/mk.conf

Add the following line at the end of the file (but above .endif):

PKG_OPTIONS.perl=       -dtrace

Now let’s build the last few dependencies:

# cd /usr/pkgsrc/13/lang/perl5
# bmake install clean clean-depends

# cd /usr/pkgsrc/13/archivers/xz
# bmake install clean clean-depends

# cd /usr/pkgsrc/09/devel/autoconf
# bmake install clean clean-depends

Compiler: from ancient to old

With this all dependencies from earlier than 2013 in place we are good to go for the biggest update. We’re still not interested in Java and Object-C, so let’s edit pkgsrc’s configuration again:

# vi /usr/local/pkgsrc/etc/mk.conf

and add one more line (e.g. after the perl one):

PKG_OPTIONS.gcc44=      -gcc-java -gcc-objc

Building the newer version of GCC means building two more dependencies as well, one of which is libgmp. GMP is the first package so far that uses C++ and in fact our C++ compiler has been broken the whole time. Luckily a symlink can heal it and another one will make GCC happy so that we can finally build it – which takes quite a bit of time (I’ve seen the compilation stop at one point and I’m not sure what happens there. But just calling bmake again will eventually complete the build process!):

# ln -s /usr/local/pkgsrc/lib/libiconv.so.7 /usr/lib/libiconv.so.7
# ln -s /usr/local/temp/gcc34/lib/libgcc_s.so.1 /usr/lib/libgcc_s.so.1
# cd /usr/pkgsrc/13/lang/gcc44/
# bmake install clean clean-depends

Once it’s build, we need to change our PATH so that the newer GCC is the primary compiler:

# mv /root/.cshrc /root/.cshrc.bak
# sed 's:temp/gcc34:pkgsrc/gcc44:' /root/.cshrc.bak > /root/.cshrc

Now all that we have to do is log out and back in:

# exit
# logout
> su -
# /usr/local/temp/bin/screen

Let’s take a look if the new compiler responds to cc (and fix c++ support along the way):

# ln -sf /usr/local/pkgsrc/gcc44/lib/libgcc_s.so.1 /usr/lib/libgcc_s.so.1
# cc -v
[...]
gcc version 4.4.7 (GCC)

GCC 4.4.7 running on FreeBSD 4.11

Yes, we really have GCC 4.4 running on FreeBSD 4.11! While it’s certainly not a modern compiler, it’s recent enough to build a lot of software. The latest release of OpenBSD, version 6.0 released on September 2016, still comes with GCC 4.2, BTW! Yes, OpenBSD maintained that all the time and heavily patch it. Still we now actually have a compiler available on FreeBSD 4.11 from 2005 which is two major versions newer!

With this we’re kind of back in business. But this post is already becoming quite long and for that reason I’m putting the “grand finale” off to one more post. See you there for the final outcome of this “little” experiment (which I hadn’t intended to write more than three posts for, but there you have it).

Updating FreeBSD 4.11 (2/4) – Digging up old graves

In part one I wrote about Legacy systems in general and showed a FreeBSD 4.11 installation for those of my readers who are interested in software history.

This post is about the first part of updating this fresh 4.11 system to a state that’s a bit less catastrophic. Remember: FreeBSD 4.11 was released in 2005 – however the ABI of each release is carved in stone with a .0 release. Which means that the software in the base system is from 4.0 and thus we venture back into the last millenium: 1999!

Initial state

To give you an idea what this means, here are a few program versions:

Various program’s versions in 4.11’s base system

So we have these programs among others:

GCC 2.95.4
Binutils 2.12.1
Perl 5.0
OpenSSH 3.5

To make matters worse, the ports tree for FreeBSD 4.11 is pretty dead, too. It’s important to get newer compilers running, but around 2005 FreeBSD used special releases to build GCC from (“gcc-core”) and I was not able to find a single mirror on the net that still holds those old and exotic files! Out of luck here. We’ll have to do without those ports.

“Modernizing” this is going to be interesting… Considering how fast the IT world moves, all of this is just as dead as it gets. So let’s prepare for the (code) smell and start digging up an old grave, shall we?

Allowing remote connection

After a passwordless login as root it makes sense to set up the right keymap (if you don’t use the default, that is). I had no idea how to do it on 4.11 and so I just gave the usual way of doing it a try – and was met with success:

# kbdmap

Looks like the keymap selection has not changed in all the time! Let’s try to make it persistent:

# echo keymap=german.iso.kbd >> /etc/rc.conf

I tried it out and it did just what I wanted. Time to try to add a regular user:

# adduser

The script is a bit different from what we’re used to today but in the end it does the same thing: Allow us to create a user. It’s important to add this user to the wheel group so that we’re able to su to root. However we need to give root a password first:

The useradd script in FreeBSD 4.11

# passwd

Ok, sshd should be running. Let’s check just to be sure:

# ps aux | grep sshd
[...]
root        75  0.0  0.1  2600 2040  ??  Is    6:26AM   0:00.11 /usr/sbin/sshd

Looks good. What about connectivity? Let’s see:

# ifconfig
vr0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        ether 00:05:5d:96:fa:f9
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
[...]

Nope, no connection. Luckily I paid attention when I the installer started from the CD and somewhere the strange-looking path of /stand caught my eye. Doing a little research, I found out that this directory used to be part of the OS and was removed in early FreeBSD 5 as it was mostly redundant with /rescue. What was in there, you ask? Have a look yourself:

# ls /stand
-sh             etc             minigzip        rm              tunefs
[               find            mount_mfs       route           usbd
arp             fsck            mount_nfs       rtsol           usbdevs
boot_crunch     gunzip          newfs           sed             zcat
camcontrol      gzip            pccardc         sh
cpio            help            pccardd         slattach
dhclient        hostname        ppp             sysinstall
dhclient-script ifconfig        pwd             test

Network configuration

There’s our friend sysinstall. I already said that it does more than just install the system. So let’s bring it up now:

# /stand/sysinstall

Sysinstall to configure the network

There we choose Networking -> Interfaces -> the appropriate NIC. No, I don’t want IPv6 and yes, DHCP is the thing.

Interface configuration using sysinstall

I’ve called the system vierelf which would be “foureleven” in English because I couldn’t think of anything better. And it’s just a test system anyway. Does the connection work now?

# ping elderlinux.org
PING elderlinux.org (212.77.232.71): 56 data bytes
64 bytes from 212.77.232.71: icmp_seq=0 ttl=54 time=24.993 ms

A little housekeeping

Alright! Let’s just take a look what services are listening right now:

# sockstat -4 -l
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      
root     dhclient   281    7 udp4   *:68                  *:*                  
root     sendmail    84    4 tcp4   *:25                  *:*                  
root     sendmail    84    6 tcp4   *:587                 *:*                  
root     sshd        75    4 tcp4   *:22                  *:*                  
root     syslogd     61    5 udp4   *:514                 *:*

Ugh! Time to make a few changes in /etc/rc.conf to deactivate any daemons except for SSH (which we need). This may not be strictly necessary but we want to improve the security of this system, right? And I wouldn’t trust those crusty old daemons at all. And whatever is not running won’t cause us any problems. So let’s get rid of them with extreme prejudice!

# vi /etc/rc.conf

To do so, add daemonname_enable=”NO” for each daemon and in case of sendmail use sendmail_enable=”NONE”.

FreeBSD 4.11 rc.conf with most daemons disabled

If you reboot now, sendmail and syslogd as well as cron, usbd and inetd will be disabled. That’s a starting point in securing the system. Let’s move on.

Replacing a dead tree

I’ll connect to the 4.11 box remotely over SSH because it’s much more convenient to have my trusty terminal at hand and to be able to copy and paste stuff:

% ssh kraileth@192.168.1.5
Unable to negotiate with 192.168.1.5 port 22: no matching host key type found. Their offer: ssh-dss

Ouch! I cannot even SSH into the system because its version of OpenSSH is so old that it only offers ssh-dss keys which have been deprecated for quite a while and disabled by default in OpenSSH >=7.0. So to connect to that old server, I have to tell my SSH client to accept ssh-dss for this connection:

% ssh kraileth@192.168.1.5 -oHostKeyAlgorithms=+ssh-dss

SSH login using “-oHostKeyAlgorithms=+ssh-dss”

Ok, I’m in. But what now? We cannot use FreeBSD’s ports tree but I strongly prefer some means of package management over installing stuff using make install. So how do we accomplish this? Enter pkgsrc. Pkgsrc is basically NetBSD’s fork of the FreeBSD ports tree. Being a NetBSD project however, it’s not limited to just NetBSD. It’s a truely portable way of building and managing software (I might write a separate post about it some time).

There’s just one problem: Downloading and decompressing the latest pkgsrc release (currently 2016Q4) won’t complete the bootstrapping process. Obviously FreeBSD 4.11 is no longer supported – which is not so much of a surprise. Time to try out older releases! After doing so I found out that 2009Q4 seems to be the last release to bootstrap successfully.

But here’s another problem: Pkgsrc doesn’t seem to keep older releases around and I also haven’t found them mirrored anywhere on the net. Pkgsrc uses CVS, however. So it’s possible to checkout older versions. FreeBSD comes with CVS as part of the base system. Unfortunately CVS works over SSH. And NetBSD’s CVS server won’t accept ssh-dss (which totally makes sense)! Since we don’t control the server, there’s also no way to just add a parameter or something to make it work. It simply doesn’t work that way.

Time to get CVS on my slightly more modern FreeBSD 11, do the checkout there and tar it all up to copy it over via scp! We’re going to get 2007Q2 instead, though, since we need things that won’t work on FreeBSD 4.11 with later versions. Oh, and if you’re not familiar with CVS, don’t worry. You don’t need to know what modules or tags are. Just copy the commands that I prepared for you and you’re good to go:

% sudo pkg install cvs
% cvs -danoncvs@anoncvs.netbsd.org:/cvsroot get -rpkgsrc-2007Q2 -P pkgsrc
% tar cvjf pkgsrc2007Q2.tbz2 pkgsrc/*
% scp pkgsrc2007Q2.tbz2 kraileth@192.168.1.5:/usr/home/kraileth

Then back on vierelf the next step is to prepare some directories and extract the pkgsrc tarball:

# mkdir -p /usr/local/temp /usr/pkgsrc
# cd /usr/pkgsrc
# mv /usr/home/kraileth/pkgsrc2007Q2.tbz2 .
# tar xvjf pkgsrc2007Q2
# rm pkgsrc2007Q2.tbz2
# mv pkgsrc 07

Bridgehead in hostile territory

Now we can bootstrap pkgsrc:

# cd /usr/pkgsrc/07/bootstrap
# ./bootstrap --prefix=/usr/local/temp --varbase=/usr/local/temp --pkgdbdir=/usr/local/temp/db

Pkgsrc 2007Q2 bootstrap complete!

Pkgsrc has been bootstrapped successfully! We just need to adjust the path variable so that the system picks up binaries from the new paths (and make those take precedence over the old system binaries). We could just change the PATH variable but it’s better to make the changes persistent. So let’s add the two new paths in the shell’s rc file in front of the others:

# vi /root/.cshrc

This is what needs to be prepended:
/usr/local/temp/sbin /usr/local/temp/bin

Root’s .cshrc file for the first phase pkgsrc

Now simply log out and become root again to have the new environment:

# logout
> su -

As a last step check if everything is right and we can access binaries in both paths:

# which bmake
/usr/local/temp/bin/bmake
# pkg_info 
bootstrap-mk-files-20061111 *.mk files for the bootstrap bmake utility
bmake-20051105nb3   Portable (autoconf) version of NetBSD 'make' utility
tnftp-20050625nb1   The enhanced FTP client in NetBSD
mtree-20070710      Utility for mapping and checking directory hierarchies
pax-20060202nb1     POSIX standard archiver with many extensions
pkg_install-20070710 Package management and administration tools for pkgsrc

Excellent! Now we have a working replacement for FreeBSD’s dead ports tree. This is definitely something that we can build upon.

Bring on some stability!

Lame pun, I know. Nevertheless it makes sense to… err… update the OS to the latest version. This is what we currently have:

# uname -a
FreeBSD vierelf.localdomain 4.11-RELEASE FreeBSD 4.11-RELEASE #0: Fri Jan 21 17:21:22 GMT 2005     root@perseus.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386

We’re on 4.11-RELEASE. The latest code for each release cycle is always in the -STABLE branch. Believe it or not: The latest code change was in April of 2014! The traditional way of getting FreeBSD code was over CVS. No, this time the problem is not ssh-dss. FreeBSD migrated from CVS to SVN (subversion) in 2008. FreeBSD CVS servers have been removed years ago. Therefore the old tools like cvsup are useless. Subversion is needed to checkout the source.

Luckily we have our pkgsrc ready. This old release has a very old port for subversion but that’s fair enough. There are a few source tarballs that are no longer available from the mirrors that pkgsrc knew for them. Not a big problem, we can download those manually:

# cd /usr/pkgsrc/07/distfiles
# fetch http://archive.apache.org/dist/httpd/httpd-2.0.61.tar.bz2
# fetch http://repository.timesys.com/buildsources/e/expat/expat-2.0.1/expat-2.0.1.tar.gz
# fetch http://download.nust.na/pub2/openpkg1/sources/DST/pkgconfig/pkg-config-0.21.tar.gz

Now we can build subversion:

# cd /usr/pkgsrc/07/devel/subversion-base
# bmake install clean clean-depends

Various dependencies will be downloaded, built and installed. Eventually subversion will be installed and available on the system. Time to tell the shell to look for new binaries and then checkout the stable source code:

# rehash
# svn co svn://svn.freebsd.org/base/stable/4 /usr/src

SVN checkout of the 4.11-STABLE code

The FreeBSD 4 base system never knew anything beyond CVS and cannot cope with the .svn directories that the svn checkout creates. World builds but the installation fails like this:

install: /usr/libdata/perl/5.00503/./.svn/text-base/Cwd.pm.svn-base: No such file or directory

Therefore it’s necessary to get rid of those disruptive directories:

# cd /usr/src
# find . -iname '.svn' -exec rm -rf {} \;

Now we can build world and kernel, install both and reboot the system:

# make buildworld
# make buildkernel
# make installkernel
# make installworld
# shutdown -r now

When the system comes back up we can SSH into it again. And there we can see that we’re on -STABLE now!

The newly built FreeBSD 4.11-STABLE

# uname -a
FreeBSD vierelf.localdomain 4.11-STABLE FreeBSD 4.11-STABLE #0: Sat Jan 28 11:53:06 GMT 2017     root@vierelf.localdomain:/usr/obj/usr/src/sys/GENERIC  i386

That’s it for today. We’re not quite there yet, but we’ve laid the groundwork for many more updates to come. Those will be described in the coming two posts of this mini series.

Updating FreeBSD 4.11 (1/4) – Blast from the past

Ah, weekend. And it’s a nice day, too: The sun shines on the snowy landscape! A perfect day to go out with the children – or to embark on a vastly different kind of adventure… Yes, you read that title right. This is not about four FreeBSD 11 machines. It’s about one FreeBSD 4.11 machine in 2017!

Legacy systems in general

Why even bother with FreeBSD 4.11 today? Well, most people would probably respect historical interest and in fact I was keen on tinkering with a legendary old 4.x release and see if you can have some fun with it (spoiler: Oh yes, you can!). But this would be a task for times when I have absolutely nothing else to do. So it’s fairly obvious that there’s another reason.

“Old habits die hard” they say and there’s definitely truth to it. Another truth, however, is that in the IT world things often don’t go away easily. There’s still a considerable number of machines out there that run DOS. In production. Doing important stuff. Sometimes just because it works sufficiently well. But often they have special tasks that for some reason or another cannot be migrated as nobody has any idea clue how you’d do this.

You want examples? Sure. The first example is “Game of Thrones” author George R.R. Martin writing his books using WordStar 4.0 on a DOS machine. A lot of companies use DOS, too, but you usually don’t hear about that. And less than a month ago a new version of FreeDOS (v. 1.2) was released.

Or do you remember the French airport that was having trouble due to using Windows 3.11? That was in 2015 and they announced that they were planning to migrate their system officially by 2017 (but expectedly by 2019 or even 2021)!

FreeBSD 4.11

One such case of a formerly very popular operating system that just won’t go away is FreeBSD 4.11. From the version number alone you can see that this is a very special one: Usually FreeBSD has three to four point releases in a release’s life cycle. FreeBSD 4 obviously was different. The main reason for that was that version 5 kind of was to FreeBSD what version 4 was for MS-DOS: Innovative in its day but disappointing especially in terms of stability (and performance) among other things.

I have to take up the cudgels for FreeBSD 5, though. It blazed a trail for important technology that we rely on today like the GEOM framework, porting the system to amd64, and so on. Implementing things like that meant digging deep into the system and doing pretty invasive changes. It totally paid off in the long run but back in the day people tended to avoid the early 5.x releases or the whole release cycle altogether.

There are many reasons why some companies still have FreeBSD 4.11 running: Critical systems that cannot be migrated, installed programs to which the source code was lost, etc. Things like that. Most of these systems are probably used internally only but I’m pretty sure that there’s also quite some 4.11 machines still serving data on the net…

Now what others do (and what their reasons for it might be) are not my primary concern – I can’t do anything about it anyway. However the company that I work for has its share of legacy systems, too. Especially younger admins hate them. Actually they are impressively stable and rarely if ever have problems. In fact they keep running, silently doing their job. But now and then you have to do some changes to them and that’s where the trouble starts: They simply cannot be compared to the modern-day Linux systems that we’re used to administer daily!

Preparations

I kept wondering if that situation could be improved. It wouldn’t be worth the effort to do this during work time but I was curious enough about the legendary 4.11 release to give it a try. There’d surely be enough to learn. And should I succeed in bringing a test system into a more modern state there would also be a benefit for us at work.

We still have all those old CDs at work but I made my decision impulsively on Saturday. Fortunately ISOs of the old Walnut Creek CD-ROMs are available for free download. Thanks, Archive.org! I burned it to a CD and was able to boot from it.

I actually installed the OS on real hardware. First on an old HP compaq nx9010 laptop that I got for free a while ago. Things worked pretty well, but since what I’m trying to do is almost all about compiling software, the compilation times proved to be a bit long. Therefore I decided to redo the previous steps on faster hardware – and since amd64 PCs are capable of running i386 programs, all was well. I just had to disable AHCI in the BIOS since FreeBSD 4.11 doesn’t know what to do with that.

The actual installation was interesting enough (for people like me who enjoy history) and therefore I decided to redo it again in VirtualBox to take screenshots and describe the installation process for the remainder of this post.

Installation

Quite some time has passed since FreeBSD 4.11 was released in 2005. More than a decade is a lot in a man’s life but it’s a hell of a lot when it comes to software and operating system development! I had never before installed any version of FreeBSD older than 7.4 and even thinking about the differences between 7.x and 11, many things changed. But what would 4.11 look like? Would I be able to install it without difficulties (I tried a very old version of OpenBSD once and quickly gave up because I didn’t really like the idea of manual partitioning of the drives…)?

FreeBSD 4.11’s Kernel Configuration Menu

I was surprised to run into something that I had never seen even before the installer started! There’s this Kernel Configuration Menu where you can choose to either boot the standard kernel or to customize it. Of course I was curious and so I decided to take a look at it.

FreeBSD 4.11 kernel drivers selection

This brings up a friendly little tool that allows for easy navigation through the driver tree. The screen is basically split into two parts: Active drivers and inactive drivers. Each one consists of several categories which are collapsed and can be expanded for a list of drivers.

Browsing FreeBSD 4.11’s additional network drivers

The standard kernel worked on both of my machines and so I didn’t play around with this after exploring it briefly. When the kernel had booted, I found myself in the good old Sysinstall utility. Yes, it’s pretty ugly and it looks a bit complicated – which is due to the fact that despite it’s name it does a lot more than just handle the installation. Yet in fact it’s really easy to navigate through.

I like the Express option that lets you perform an installation as quickly as possible by asking you only the absolute minimum of necessary questions. This goes as far as not asking for a root password and leaving it blank on your first login! Fair enough, it’s easy to set one yourself. Current versions of FreeBSD do not have this option. That may be due to the fact that it’s not terribly helpful for setting up a serious system. And the use case that I have for it here is definitely more of an exception!

FreeBSD 4.11’s Sysinstall: Main Menu

The first thing that’s essential is of course disk partitioning. If you don’t require any special setup, you can simply use the whole disk. I was glad to see that this can be done by simply pressing “A”!

Installing FreeBSD 4.11: Disk partitioning

Just like today FreeBSD 4.11 allows you to choose if you want to install a boot manager, just write the boot loader or don’t do anything. The first option can be used for dual-boot systems where you want to load either FreeBSD or another OS. The last option won’t work by itself; if you choose that, you need to work with an external boot manager that supports FreeBSD. In my case just installing the standard loader makes most sense.

Installing FreeBSD 4.11: Choosing boot options

Then there’s creating the disklabels. This is another thing where I’m happy to find an auto defaults option (simply pressing “A” again) because I just wanted a quick test system anyways and didn’t want to think too much about the disklabel layout!

Installing FreeBSD 4.11: Editing disk labels

Finally we have to select what to install. To make things easy, sysinstall offers several distribution sets to choose from. Minimal is all that I need for my experiments, but back in the day a lot of the other options were definitely quite handy. Two things stand out: You can choose to install FreeBSD + X11 for example. Current versions of FreeBSD don’t let you do that. You’ll always get a console only system and have to install X11 yourself (which IMO is something that leaves the barrier unnecessarily high for new users who don’t know FreeBSD nor its packaging system, yet!).

Sysinstall also allows for custom distribution selection. This gives you access menu that allows for really fine-grained choices. Again, FreeBSD’s installer cannot do that today. While it’s probably nice to have that option, it’s not a must-have. People who have use cases for that will probably know what they do and install FreeBSD manually, anyway. Still I like having it in the installer!

Installing FreeBSD 4.11: Choosing distribution sets

The only thing left is selecting what medium to install from – the CD in my case. Formatting a large drive takes quite a moment because we’re talking UFS1 here! Don’t start looking for a way to use UFS2, it’s not available in FreeBSD 4. The actual “minimal” installation is quite quick and just a moment later I can reboot and the system comes up just like you’d expect it to. Now that was the easy part of my adventure. Next stop: Brushing the dust away and updating that beast(ie)!