Illumos (v9os) on SPARC64 SunFire v100

Over the last month or so I’ve written a couple of articles on an old SunFire v100 machine that I own for a while now. First I took a look at the hardware of the machine and the LOM (Lights Out Management). Then I installed OpenBSD 6.0 from CD and updated all the way to 6.5. Finally I played a bit with OpenBSD to see what it can do and how well it supports SPARC64. This post will be the last SPARC64 one before I visit other topics again.

v9os?

While I was pretty happy with OpenBSD on the SunFire, there’s one reason that I wanted to try out something else, too. That reason has three letters: Z-F-S. The first thing that I tried out when I got the hardware, was FreeBSD – but I ran into problems. I’ve managed to overcome circumvent them (might be worth another story in the future), only to find that FreeBSD does not support ZFS on SPARC64!

One option that suggests itself, is just putting Solaris on there. I have a copy of Solaris 10 for Sparc, but I prefer to keep things Open-Source. Also there’s the problem, that my machine is old enough to not have a DVD drive and it doesn’t support booting from USB and the like.

So it’s illumos. Since I’m really just getting started with the broader Solaris universe, I had to do a little research first. And I was a little surprised that most illumos distros seem to not even support Sparc at all! Of the four that do

  • OpenSXCE seems dead (last release in 2014)
  • DilOS uses Debian packaging (which is not my cup of tea at all)
  • Tribblix sounds really interesting to me, but does not fit on a CD
  • v9os is a minimal Sparc distro that is small enough

As you can see, there wasn’t so much choice after all! While v9os is an experimental one-man project that you should probably stay away from for production use, it might be just right for my purposes of tinkering with an old machine.

Installing the OS – first try

There are not many preparations necessary: I downloaded the ISO image and burned it on a CD. Then I connected to my SunFire via serial, powered it on and put the CD into the drive. It takes quite some time, but after a while I can read that v9os is in fact starting.

Booting up v9os from the CD

After the system booted, it gives the user the option to select a keymap.

Keymap selection

Then it shows the installation menu. There you can choose if you want to install, load additional drivers, drop to a shell, change the terminal type or reboot. I go with the first option.

v9os installation menu

After a moment the installer has started an a welcome screen is printed. Unfortunately in my case there’s a problem with the CD, so that four lines of debug info overwrite important information: How to actually proceed with the installation! But this is an OpenSolaris derivative, and so it’s not that hard to figure out that F2 is the key to go on.

v9os installer: Welcome screen

Next it’s selecting the disk to install on. I thought that it all looked good – and didn’t pay much attention to the message “A VTOC label was not found.”. VTOC is the Volume Table Of Contents, the SPARC partition scheme (think MBR/GPT on amd64). We’ll come back to that a little later. 😉

v9os installer: Disk selection

I think that the installer is quite nice. It even offers help pages that give newcomers like me an idea of what they should do for the current step. Great work on that!

v9os installer: Disks help page

Then you can choose to either dedicate the whole disk to v9os or just use a slice. I decide to go the easy route and select the former.

v9os installer: Disk layout selection

Now the installer wants to know the hostname for the new system. The suggested default of v9os is fine for me since I don’t plan to add another machine with that OS to my network anytime soon.

v9os installer: Hostname selection

Finally you can select the time zone – or rather: the zone region.

v9os installer: Time zone selection

Unfortunately things went sideways after that choice and I had to reset the machine…

Ok, after going through the previous steps again, I decided to give the advanced setup a try and selected slicing up the drive.

v9os installer: Slice selection

Unfortunately the result was the same as before: The installer just died. I tried again a few times, playing with different slice setup, but didn’t have any luck.

The installer died… Time to reboot.

At this point I was out of ideas on what else I could try, so I removed the CD and powered down the system.

Writing the label manually

When I powered the system on again, I had forgotten that I removed the CD and to my surprise OpenBSD (the system that I had previously installed on the machine) booted up! This meant that the installer had not even changed anything on the disk, yet!

My next guess was (and still is) that the v9os installer might have problems with BSD disklabels being present on the drive. I took a look at the disklabel from OpenBSD, just to find out some information about the drive.

OpenBSD’s disklabel information of the system hard drive

Then I booted the v9os install medium again but this time selected the shell option. After a little research I found out how to get some drive information on Solaris with iostat.

v9os shell session: Collecting drive hardware info

Next I decided to give the format utility a try. I don’t know if v9os stripped down some hardware information and that together with the disk being really old, it wasn’t properly auto-detected. So I had to do something that I haven’t done in years (and never missed it): Typing in the geometry information by hand!

Typing in disk geometry information (Ah, the (bad!) memories…)

Once the drive has been described to the utility, it shows a menu of what it can do. I haven’t used that program before and judging from the name alone was a bit surprised at how powerful it seems to be. Things like being able to define profiles must have been pretty useful in the past.

Solaris’ format utility

Since I want to partition the drive, I select that. I’m presented with a sub-menu, giving me some more choices.

Partitioning menu of format

I have no clue what a Solaris partitioning scheme should look like (need to explore some older versions of that OS somewhen!).

Partitioning the drive for Solaris

So I look around a little but eventually accept the proposed default and just hope that this works.

Installing the OS – second try

After restarting the machine again and choosing the installer, it looks like this time there is no missing disklabel. At least! But will it make a difference?

Returning to the installer: Partitioning was detected

And yes! Now the installer continues and gets the data written to disk!

Finally installing the OS!

The process takes quite a while – but that’s due to the slow machine that I’m using. Eventually the installation is finished.

v9os installer: All done!

First steps with v9os

Another reboot and after removing the CD-ROM from the drive, the freshly installed system boots up. A moment later it displays the prompt where I can log in using the user root and the password solaris.

First start of v9os

The first thing that I want to do is to get rid of the serial console. So I set up networking and enable SSH.

Setting up networking and enabling SSH

Then I disable the automounter to make the home directory writable and create a user for remote SSH login. Finally I enable the machine to do name resolution and give the new user a password.

Adding a user and name resolution capabilities

That should suffice to SSH into the box from another machine.

Package management with IPS

Logging in remotely works just fine. As v9os does not have an online package repository, I have to download a compressed copy of the repository from SourceForge.

SSHing into the v9os box and downloading the package repository

I don’t know much about the IPS package system and thus really struggle to make it all work. There is no guide on the v9os site and so I try to put the downloaded file in various locations, decompress it and try everything again. Since that also doesn’t work, I unpack the contents of the archive but still cannot get it right…

Struggling to get the repo working…

After more than an hour of struggling with pkg, reading manpages, doing online research and trying to fit everything together, I finally manage to remove the default publisher that comes with the system and add a new one that eventually works!

Finally figured out how to deal with IPS publishers

The v9os operating system is one of the strangest Unices that I’ve ever touched in not providing the vi editor with the system! But now that I have the repository available, I can simply install vim to find out that using packages does work after all.

Installing packages (vim) works!

This is about how far I wanted to take this quick post on v9os. If I had a faster machine, I might have been tempted to try and build the system from source. But with my old SunFire… No.

While v9os might not be fit for production use, I accomplished one goal over OpenBSD: I have an operating system on the machine that is installed on ZFS!

ZFS on SPARC64 with v9os

Conclusion

The v9os operating system is an exotic one for sure. But it’s nice to see that somebody values SPARC64 machine and illumos enough to put the time required to built something like this into such a project. And actually I think it’s not half bad! I didn’t do too much with it, but it seemed stable and except for the installer problem (it would probably just have worked on an empty drive) everything worked fine.

Well, maybe some hints on how to get the package repo in place would have saved me some time… On the other hand Solaris veterans are likely to get it working with just a few commands. And while it has been kind of frustrating for a while, it has also lead to at least a basic understanding of what IPS is and how it works. I’m sure that I’d have missed at least some of that if I had just copied some lines from a guide.

I might not end up making v9os my primary operating system (for various obvious reasons). But it’s another nice little part in the mosaic of the illumos world that I’ve started exploring. Also I noticed that I’ve become a little bit more comfortable with using an OpenSolaris-derivative. Compared to my first encounter with OmniOS, it didn’t take me as long to figure out the very basics again. Which is always a good sign.

Running OpenBSD on SPARC64 (HTTPd, packages, patching, X11, …)

In my previous post I described the process of installing OpenBSD 6.0 on a SPARC64 machine and updating it all the way to 6.5. Now it’s time to actually do something with it to get an idea of how well OpenBSD works on this architecture!

OpenBSD’s base system

The OpenBSD team takes pride in providing an ultra-secure operating system. It’s a well-known fact that the project’s extremely high standards only apply to the base system. Every now and then critics pop up and claim that this basically defeats the whole idea and even accuse the project of “keeping their base system so small that it’s useless by itself” to keep up their defined goals.

There’s some truth to it: The base system is kept (relatively) small if you compare it to some of the fatter operating systems out there. But that’s about it because actually these allegation could not be further from the truth. The base system includes doas, a simpler sudo replacement. It comes with tmux. OpenBSD even maintains it’s own fork of X.org, called Xenocara (not even FreeBSD comes with an X11 server by default) and there’s in fact a lot that you can achieve with the base system alone! Let’s look at one such possibility.

HTTPd

Since the OpenBSD developers are convinced that a webserver is something to keep around all the time, there’s one in base. Originally they used the Apache HTTPd for this. The problem was that at some point, the Apache Foundation decided to give up their Apache 1.0 license and replace it with version 2.0 (they had been criticized a lot for being incompatible with the GPL and the new version solved that problem). The newer version also made the license less simple and permissive than it had been before and OpenBSD did not like the new license. For that reason they basically stayed with the old Apache 1.3 webserver for a long time. They maintained and patched it all that time, but the software really begun to show it’s age.

So for version 5.6, OpenBSD finally removed the old Apache webserver in base and replaced it with Nginx. One release later, they did away with that, too, because they felt that it was starting to become too bloated for their needs. They imported OpenBSD HTTPd into base instead: A home-grown, very simple webserver. It evolved over time, but even though it having gotten more features implemented and becoming a fine little webserver, it strives to keep it simple.

The developers resist the temptation to add new features just because they could and have even made a list of things that some people might want which however will never be implemented because they would raise complexity to an unacceptable level. OpenBSD HTTPd does not want to be a webserver for everyone. It wants to be a ultra-secure webserver that does enough to be useful to many people. If you have any needs above what it offers – get another one.

Simple static website configuration of OpenHTTPd

The simplicity of HTTPd adds a lot to its beauty. I’ve written some HTML for a test page (see screenshot). All of the configuration that I need to do for HTTPd is as follows:

server spaffy.local {
    listen on egress port 80
}

Yes, that’s all that is required: I basically define a vHost (“Server” in HTTPd lingo) and have the application listen on the HTTP default port 80 on egress (a keyword which means whatever interface has the default route). Let’s check if that configuration really is valid by issuing

httpd -n

And it is! Impossible? No. Remember that OpenBSD comes with sane defaults. For that reason there’s usually pretty little that you need to configure. You could, of course. And we’ll be doing that a little later.

Now let’s force-start httpd (we need -f since the service is not enabled, yet, and we want to manually start it once):

rcctl -f start httpd

I’ve edited the /etc/hosts file on my laptop to be able to use the spaffy.local name. So now I can just type that into the address bar of my browser and reach the test page that the SPARC64 machine hosts. OK, a static page probably doesn’t impress you so much. Fortunately that’s not all that we can do in just relying on what base offers!

Static test page displayed in browser

CGI

OpenBSD also comes with Perl as part of the default install. I got that Lama book several years ago, read through about 2/3 of it and then decided that I didn’t like Perl too much. For that reason I never really did anything with it, but here I want to do something with what OpenBSD provides me with, so Perl is a logical choice and I might finally do something with it. Here’s what I came up with:

#!/usr/bin/perl
use strict;
use warnings;

my $osname = `uname -s`;
my $osver = `uname -r`;
my $osarch = `uname -m`;
chomp($osname, $osver, $osarch);

my @months = qw( Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec );
my @days = qw( Sun Mon Tue Wed Thu Fri Sat Sun );
my ( $sec, $min, $hour, $mday, $mon, $year, $wday, $yday ) = gmtime();

print "Content-type: text/html\n\n";
print "<html><head><title>Greetings!</title></head>";
print "<body>Hello from <strong>$osname $osver</strong> on <strong>$osarch</strong>!";
print "<br><br>This page was created by Perl $^V on $days[$wday], $months[$mon] $mday";

if (length($mday < 2)) {
  if (substr($mday, -1) == "1") {
    print "st"; }
  elsif (substr($mday, -1) == "2") {
    print "nd"; }
  elsif (substr($mday, -1) == "3") {
    print "rd"; }
  else {
    print "th"; }
} else {
  if ((substr($mday, 0, 1) ne "1") and (substr($mday, -1) == "1")) {
    print "st"; }
  if ((substr($mday, 0, 1) ne "1") and (substr($mday, -1) == "2")) {
    print "nd"; }
  if ((substr($mday, 0, 1) ne "1") and (substr($mday, -1) == "3")) {
    print "rd"; }
  else {
    print "th"; }
}

print ", $hour:";
if (length($min) == 1) {
  print "0";
}
print "$min (UTC)</body></html>";

Nothing too fancy, but for a first attempt at writing Perl it’s probably OK. After making the script executable, I can run it on the system and get the expected output. Things get a little more complex, though. HTTPd runs in a chroot for security reasons. And just copying the script into the chroot and trying it to execute in a chrooted environment fails with “no such file or directory”.

Huh? I just copied it there, didn’t I? I sure did. The reason for this happening is that the Perl interpreter is not available in the chroot. So let’s copy that one over as well and try again. Abort trap! How do they say? Getting a different error can be considered progress…

Perl CGI script failing due to chroot

Ok, now Perl is there, but it’s not functional. It requires some system libraries not present in the chroot. Using ldd on the Perl executable, I learn which libraries it needs. And after providing them, I can run the script in the chroot! There is a new problem, though: Perl is complaining about missing modules. The simplest solution in our case is to just remove them from the demo script as they are not strictly (haha!) necessary.

Providing Perl dependencies in the web chroot

On to the next step. Here’s a little addition to the HTTPd configuration:

    location "/cgi-bin/*" {
        root "/"
        fastcgi
    }

It basically adds different rules for the case that anything below /cgi-bin is being requested. It changes the document root for this and enables fastcgi. Now I only need to start the slowcgi service (OpenBSD’s shrewdly named fastcgi implementation) and restart HTTPd. My Perl program makes uses of the system’s uname command, so that should be made accessible in the chroot, too, of course.

Finishing the dynamic webpage setup

And that’s it. The script is executed in the webserver and the expected resulting page generated ,which is then served properly:

Dynamically created webpage displayed in beowser

I think this is pretty cool. Try to do that with just the default install of other operating systems! BTW: Want to make HTTPd and slowcgi start automatically after boot? No problem, just put the following into /etc/rc.conf.local:

httpd_flags=""
slowcgi_flags=""

This makes the init system start both daemons by default (and you can of course drop the “-f” flag to rcctl if you need to interact with them).

Binary packages

For OpenBSD 6.5, pre-built packages are offered for 8 of the 13 supported architectures – among them SPARC64. There’s just a couple short of 9,500 packages available (on amd64 it’s 10,600 – so in fact most packages are there)!

Things like GCC 8.3 and even GNAT 4.9 (the Ada part of GCC which is interesting because it’s written in Ada and thus needs to be bootstrapped to every new architecture by means of cross-compiling) are among the packages, as is LLVM 7.0. When it comes to desktop environments, you can choose e.g. between recent versions of Xfce, MATE and Gnome.

Actually, SPARC64 is one of only 4 architectures (the others being the popular ones amd64, i386 and arm64) that are receiving updates to the packages via the packages-stable repository. In there you’ll find newer versions of e.g. PHP, Exim (which had some pretty bad remote exploits fixed), etc.

Basic OpenBSD package management

I choose to install the sysclean package. Remember when I said that I skipped deleting the obsolete files when updating the OS in my last post? This program helps in finding files that should be deleted. However it’s not too intelligent – it just compares a list for a fresh system to the actual system on disk. For that reason it also lists a lot of files that I wouldn’t want to delete. Still it’s helpful to find out obsolete files that you might have forgot to remove.

Sysclean shows a lot of possible remove candidates

Errata patches

While OpenBSD tries it’s very best at providing a safe to use operating system, there really is nothing both useful and free from errors in the IT. If problems with some component of the system are found later, an erratum is published for it. If you are using OpenBSD in production, you are supposed to keep an eye on errata as they are released. Usually they consist of a patch or set of patches for system source code as well as instructions on how to apply it and recompile the needed parts.

Since version 6.1, OpenBSD comes with a handy utility called syspatch(8), which can e.g. be used to fetch binary patches for all known errata that have not been applied to the OS on the respective machine. This is nice – but it’s only available for amd64, i386 and arm64. So on SPARC64 we still have to deal with the old manual way keeping the system secure. However errata patches are also applied to the -STABLE branch and we can use that to get all the fixes.

No syspatch on SPARC64 – tracking -STABLE manually as it used to be

To upgrade our installation to 6.5-STABLE, the first step is to get the operating source of the current release (the sys tarball contains the kernel and src the rest of the base system). After extracting those, CVS is used to update the code to the latest 6.5-STABLE.

Done getting the stable changes from CVS

Once that’s done, it’s time to build the new (non-SMP) kernel:

# cd /sys/arch/$(machine)/compile/GENERIC
# make obj
# make config
# make && make install
# reboot

Building a 6.5-STABLE kernel

On my SunFire v100 the kernel build took 1h 20m. I was curious enough to build the userland as well, just to see how long it would take… The answer is: 85h 17m! I think that LLVM alone took about three days. The rest of the system wasn’t much of a problem for this old machine, but LLVM certainly was.

BTW, I had problems with “permission denied” when trying to “make obj”. After reading the manpage for release(8), I found out that /usr/obj should be owned by build:wobj with 770 permissions which had not been the case on my system.

Kernel build complete

Having done that, I thought that I might build Xenocara as well, to compare how long it takes to build. So I got the sources for that, too, updated them via CVS and built it. It took 9h 26m to build and install.

Xenocara built from (-STABLE) source

X11 on SPARC64

I had left out all X11-related distribution sets when installing OpenBSD. But after having installed Xenocara from source, I had it all available. So I decided to just do something with it. Since the server does not have a graphics card, I cannot run any X program on it directly, because the xserver won’t run. I decided to get a graphical application that is not part of Xenocara installed first. After browsing through the list, I settled on Midori, a WebKitGTK-based webbrowser.

Installing the Midori browser via packages

It took a moment to install all the dependencies, but everything worked. As the next step I enabled SSH X11 forwarding and restarted SSH.

Midori is installed, allowing X11 forwarding for SSH

After connecting to the SPARC64 machine via SSH and checking that the DISPLAY environment variable was set, I could just launch Midori and have it sent over to my laptop that I used to SSH into the other box. So the browser is being executed on the SPARC64 server but displayed on my other machine.

SSHing into the SPARC64 machine and forwarding Midori to my amd64 laptop

Everything worked well, I could even visit the OpenBSD homepage and it was rendered correctly.

The webkit-based browser works well on SPARC64!

Conclusion

OpenBSD is a fine operating system for people who value quality. The SPARC64 port of it seems to be in pretty good shape: Most packages and even stable-package updates are available. What is missing, is syspatch support – but only three architectures have that right now. Also the system compiler is still the ancient GCC version 4.2 which was the last one before the project switched the license to GPLv3.

OpenBSD 6.6 has been released one day after I finished compiling 6.5-STABLE. On amd64 I could now use sysupgrade(8) to upgrade to the new release even easier than before. This is also not supported on SPARC64. But these two little shortcommings just mean a little extra work that all OpenBSD users on any platform had to do anyway until not that long ago.

For 6.6 there are even more packages available for SPARC64. E.g. the Rust compiler has been bootstrapped on this architecture which definitely is great news. Maybe the system compiler will change to LLVM/Clang one day, too. Right now the SPARC64 backend for Clang is incomplete upstream at the LLVM project, if I understood things right. But we’ll see. Maybe it’ll become available in the future. I guess I’ll really have to get a newer SPARC64-based machine with a faster processor. Luckily OpenBSD supports quite a few of them.

OpenBSD on SPARC64 (6.0 to 6.5)

Earlier this year I came by an old SunFire v100 that I wrote about in my previous article. After taking a look at the hardware and the LOM, it’s time to actually do something with it! And that of course means to install an operating system first.

OpenBSD

OpenBSD, huh? Yes, I usually write about FreeBSD and that’s in fact what I tried installing on the machine first. But I ran into problems with it very early on (never even reached single user mode) and put it aside for later. Since I powered up the SunFire again last month, I needed an OS now and chose OpenBSD for the simple reason that I have it available.

First I wanted to call this article simply “OpenBSD on SPARC” – but that would have been misleading since OpenBSD used to support 32-bit SPARC processors, too. The platform was just put to rest after the 5.9 release.

OpenBSD 6.0 CD set

Version 6.0 was the last release of OpenBSD that came on CD-ROM. When I bought it, I thought that I’d never use the SPARC CD. But here was the chance! While it is an obsolete release, it comes with the cryptographic signatures to verify the next release. So the plan is to start at 6.0 as I can trust the original CDs and then update to the latest release. This will also be an opportunity to recap on some of the things that changed over the various versions.

Preparations

I had already prepared the machine for installation previously, so I only had to make a serial connection and everything was good to go. If you’re in need of doing this and don’t feel like reading the whole previous article, here’s the important steps:

  1. Attach power to go to the lom prompt
  2. Issue boot forth and then poweron to go to the loader
  3. At the ok prompt use setenv boot-device cdrom disk to set the boot order
  4. Set an alias for the CD-ROM device with nvalias cdrom /pci@1f,0/ide@d/cdrom@3,0:f
  5. Reset the machine with reset-all or powerdown and then poweron again

Booting up the OpenBSD 6.0 sparc64 CD

Insert the OpenBSD installation CD for SPARC64 and after just a moment you should be in the installation program.

Installing 6.0

OpenBSD’s installation program is very simple. It’s basically an installation script that asks the user several questions and then goes ahead and does the things required for the desired options. In the Linux world e.g. Alpine Linux does the same, and I’ve always liked that approach.

OpenBSD 6.0 installer started

On a casual installation, the script would ask for the keyboard layout. But since we’re installing over serial here, that doesn’t matter. It asks for the kind of terminal instead. Since our CPU architecture is SPARC64, OpenBSD assumes we’re using a Sun Terminal. Well, I don’t, so I choose Xterm.

Of course we need a hostname for the new system. Since it’s Puffy (the OpenBSD mascot) on SPARC here, I settled on spaffy. 😉

Choosing the root password

Next is network configuration. DHCP is fine for this test machine. Then the root password is being set.

Of course I want to access the box over SSH later, so that I don’t need the serial connection anymore and can put the machine in a different room. Compared to many x86 servers it’s not as loud as those, but still quite a bit louder than you would want a machine sitting directly next to you to be. Allowing root over SSH is very bad practice, so I create a user next and disallow remote root logins.

Selecting the partitioning

Then I choose my timezone. Next is deciding on the partitioning. There I noticed a difference compared to i386/amd64 installations. I have a habit of creating partition B first (to put the swap space on the beginning of the drive). When I tried to do this, the installer told me that this architecture didn’t allow doing that. I assume that limitation is due to Sun’s partitioning scheme VTOC that is being used on the SPARC machines. So I created them in order.

What you can see on the screenshot is OpenBSD’s default partitioning. It’s more complex than many people may be used to, but for a good reason. Remember that you can mount filesystems with different options? That way you can e.g. have /tmp mounted noexec. OpenBSD makes good use of this, e.g. enabling or disabling W^X protection on a filesystem-wide base. This is not a production machine, though, and the drive is fairly small for today’s needs. So in the end I went with a much simpler way of dividing the drive.

Selecting the distribution sets to install

Finally I need to choose what to install. OpenBSD offers so-called “sets” for various parts of the full operating system. Since I’m only installing 6.0 as a starting point, I go with the minimum required options: The kernel (bsd) and the base system.

I have no use for the install (ramdisk) kernel (bsd.rd) or the SMP-enabled multi processor kernel (bsd.mp). Also I don’t need the system compiler (comp), manpages (man) or small games (game). Of course I also don’t need the X11-related sets.

Installation finished!

Then the installer goes off and prepares everything. When it has finished, the only thing that is left is rebooting the system (and removing the CD). Now we can also change the boot order in the ok prompt, to set it to booting from disk only, speeding up the boot time minimally:

ok> setenv boot-device disk

And that’s it! Now I have an old but known good version of OpenBSD on my SunFire box.

Freshly installed OpenBSD 6.0 booted up

Updating to 6.1

Alright. What’s next? Running a 3 years old version of OpenBSD is probably not that good an idea if newer versions are available for this architecture – and they are.

So the first thing to do is fetching the ramdisk kernel of version 6.1 and the signature for it. Then I check the integrity of the kernel with signify(1). Everything is fine, so I go on and replace the standard kernel with the install kernel for the newer version. There’s probably a better way to do this, but the SPARC bootcode seems to have “bsd” as the kernel file name hard-coded and I admittedly didn’t dig very deep to figure out a different way of booting alternate kernels.

Getting 6.1 ramdisk kernel and verifying signature

After restarting, the systems boots into the install kernel. This time I select upgrade instead of install, of course. The installer then checks the existing operating system (or at least the root partition).

I then select http for the location of the sets and point the installer to a mirror that still holds the old releases.

Installer started in upgrade mode

Next is selecting the distribution sets to be installed. Again I choose only the bare minimum, since the upgrade is just an intermediary step to upgrading all the way to a current release.

In earlier versions of OpenBSD, etc was a separate set. Since the files required to check newer releases are in /etc, I’d have chosen a different installation strategy if they were still available separately. However the etc set has been included in the big base set for a while now.

Necessary sets updated

After the sets have been downloaded and extracted the upgrade is mostly complete. The remaining things are done in the live system. So it’s time to complete this step and reboot.

Configuration files get updated on first boot after the OS upgrade

OpenBSD automatically updates various configuration files for the new release. If you pay attention, you’ll see that there is one case where the changes could not be merged automatically. So we will I need to see to that myself.

The system also looked if newer firmware was available. However this was not the case (which really is no wonder on this old machine).

Merging OpenSSH config and adding installurl

After doing the manual merge of the OpenSSH configuration, it’s time to do the final tasks to complete the upgrade. OpenBSD keeps a detailed upgrade guide for each version that lists the required manual steps. In fact you should read it before doing the upgrade, since it can involve steps that need to be done prior to booting the install kernel and updating the base system! I skipped them, because they didn’t apply in my case – e.g. I hadn’t installed the manpages anyway.

I chose to only set the installurl since that one is really convenient. Actually I should remove some obsolete files from the filesystem, too. But I decided to leave this for later as there is another method to do so.

Updating to 6.2

Getting the system updated to 6.2 means repeating what I did for the 6.1 update: Get the ramdisk kernel for the new release as well as the signature and verify it. Once that’s done, another reboot is in order.

Downloading and preparing OpenBSD 6.2 install kernel

One thing that’s different is that the installer now defaults to fetching from the web and not from CD. And thanks to setting the installurl before I rebooted, it also knows the default mirror to get the sets from. Which makes the process of upgrading even more straight-forward and convenient.

OpenBSD 6.2 installer: Now knows the URL to fetch from

Finishing the upgrade after the actual unpacking of the new files takes a bit longer for this version. After making all known device nodes, the installer re-links the kernel! This is due to a new feature called KARL (Kernel Address Randomized Link). The idea here is that the objects that make up the kernel are linked in random order for each reboot, essentially creating a new and unique kernel every time. This makes it much harder or even impossible to use parts of the kernel otherwise known to be in certain memory regions for sophisticated attacks.

OpenBSD 6.2 introduced Kernel re-linking (“KARL”)

Oh, and did you notice that the bsd.mp set is gone? This machine only has a single-core CPU and therefore the SMP kernel doesn’t make much sense. The installer detected the CPU and did not offer to install the SMP kernel (even though it of course is still available for machines with multiple cores).

As always, the system needs to rebooted after the upgrade is complete. Just a moment later I’m greeted by my new OpenBSD 6.2! Again I’m skipping the manual steps to be taken afterwards.

OpenBSD 6.2 booted up

Updating to 6.3

Preparing and doing the upgrade for 6.3 is just like you’ve seen twice now, so I’m not going to repeat it. There’s one new feature in the installer that could be mentioned, though: After the upgrade is complete, the reboot option is now the default thing that the installer offers instead of just dropping you to a shell. This means you can save another 6 keystrokes when updating! Yay! 😉

OpenBSD 6.3 install kernel: Rebooting after completion is now the default choice

Updating to 6.5

The upgrade to 6.4 is simply more of the same. Of course I did that step, but I’m cutting it out here. 6.5 is the most recent release as I’m writing this (though 6.6 is already around the corner). This means I’m going to do one more upgrade, following the process that we know pretty well by now: Get and verify bsd.rd, boot it and select “Upgrade”.

Choosing all the sets except for X11-related ones for 6.5

This time I decide to install all the sets except for anything X11-related. The SunFire v100 is a server-class machine which does not even have a graphics card! For that reason there’s no VGA port to connect a monitor to, either. And while X11 could still be of some use, it’s simply not needed at all.

Upgrade to OpenBSD 6.5 complete

Again the upgrade process takes a bit longer, but that’s only thanks to the additional sets (as well as the base distribution getting a little bigger and bigger with each release). After just a little while everything is done and there’s one more reboot to make.

OpenBSD 6.5 booted up and ready

All done! I now have a fine OpenBSD 6.5 system up and running on my old SPARC64 box. And even better: Everything has been cryptographically verified to be the data that I want and no bad person has tempered with it. Sure, the system has not been cleaned up, yet – and it’s just 6.5-RELEASE with no errata fixes applied. Still I’d say: We’re off to a good start! Aren’t we?

What’s next?

In the next post I intend to explore the system a little and find out where there are differences from a common amd64 installation of OpenBSD.

A SPARC in the night – SunFire v100 exploration

While we see a total dominance of x86_64 CPUs today, there are at least some alternatives like ARM and in the long run hopefully RISC-V. But there are other interesting architectures as well – one of them is SPARC (the Scalable Processor ARChitecture).

This article is purely historic, I’m not reviewing new hardware here. It’s more of a “20 years ago” thing (the v100 is almost that old) written for people interested in the old Sun platform. The intended audience is persons who are new to the Sun world, who are either to young like me (while I had a strong interest in computers back in the day, I hadn’t even finished school, yet, and heck… I was still using Windows!) or never had the chance to work with that kind of hardware in their professional career. Readers who know machines like that quite well and don’t feel like reading this article for nostalgic reasons might just want to skip it.

The SPARC platform

SPARC is a Reduced Instruction Set Computing (RISC) Instruction Set Architecture (ISA) developed by Sun Microsystems and Fujitsu in 1986. Up to the Sun-3 series of computers, Sun had used the m68k processors but with Sun-4 started to use 32-bit SPARC processors instead. The first implementation is known as SPARCv7. In 1992 Sun introduced machines with v8, also known as SuperSPARC and in 1995 the first processors of SPARCv9 became available. Version 9, known as UltraSPARC, is a 64-bit architecture that is still in use today.

SunFire v100: Top and front view

SPARC is a fully open ISA, taken care of by SPARC International. Architecture licenses are available for free (only an administration fee of 99$ has to be payed) and thus any interested corporation could start designing, manufacturing and marketing components conforming to the SPARC Architecture. And Sun did really mean it with OpenSPARC: They released the Verilog code for their T1 and T2 processors under the GPLv2, making them the first ever 64 bit processors that were open-sourced. And not enough with that – they also released a lot of tools along with it like a verification suite, a simulator, hypervisor code and such!

After Sun was acquired by Oracle in 2010, the future of the platform became unclear. Initially, Oracle continued development of SPARC processors, but in 2017 completely terminated any further efforts and laid off employees from the SPARC team.

Fujitsu has made official statements that they are continuing to develop the SPARC-based servers and even about a “100 percent commitment”. In the beginning of this year, they even wrote about a Resurgence of SPARC/Solaris on the company’s blog and since they are the last one to provide SPARC servers (which are still highly valued by some customers), chances are that they will continue improving SPARC. According to their roadmap, even a new generation is due for 2020.

So while SPARC is not getting a lot of attention these days, it’s not a dead platform either. But will it survive in the long run? Time will tell.

SunFire v100

I’m working for company that offers various hosting services. We run our own data center where we also provide colocation for customers who want that. Years ago a customer ran a root server with an (now) old SunFire v100 machine. I don’t remember when it was decommissioned and removed from the rack, but that must have been quite a while ago.

SunFire v100: Back view

That customer was meant to come over to collect the old hardware and so we put the machine in the storage room. For whatever reason, he never came to get it. Since it had been sitting there for years now, I decided to mail the customer and asked if he still wanted the machine. He didn’t and would in fact prefer to have us to dispose of it. So I asked if he’d be ok with us shreddeing the hard drives and me taking the actual machine home. He didn’t have any objections and thus I got another interesting machine to play with.

The SunFire v100 is a 1U server that was introduced in 2001 and went EOL in 2006. According to the official documentation, the machine came with 64 bit Solaris 8 pre-installed. It was available with an UltraSPARC IIe or IIi processor and had a 40 GB, 7200 RPM IDE HDD built-in. My v100 has 1GB of RAM and a 550 MHz UltraSPARC IIe. I also put a 60 GB IBM HDD into it.

It has a single PDU, two ethernet ports as well as two USB ports. It also features two serial ports – and these are a little special. Not only are they RJ-45, but they have two different uses cases. One is for the LOM (we’ll come to that a little later), the other one is a regular serial port that can be used e.g. to upload data uninterrupted (i.e. not going to be processed by the LOM). The serial connection uses 9600 baud, no parity, one stop bit and full duplex mode.

RJ-45 to DB9 cable and DB9 to USB cable

The other interesting thing is the system configuration card. It stores host ID and MAC address of the server as well as NVRAM settings. What is NVRAM? It’s an acronym for Non-Volatile Random-Access Memory, a means for storing information that must not be lost when the power goes off like regular RAM does. If you’re thinking “CMOS” in PC terms, you’re right – except it seems that Sun used a proper means of NVRAM and not an in fact volatile source made “non-volatile” by keeping the data alive with the help of a battery. The data is stored on a dedicated chip, or in this case on a card. The advantage of the latter is that it can be easily transferred to another system, taking all the important configuration with it! Pretty neat.

Inside the v100

When I opened up the box, I was actually astonished by how much space there was inside. I know some old 1U x86 servers from around that time (or probably a little later) that really are a pain to work with. Fitting two drives into them? It’s sure possible, but certainly not fun at all. At least I hated doing anything with them. And those at least used SATA drives – I haven’t seen any IDE machines in our data center, not even with the oldest replacement stuff (it was all thrown out way before I got my job). But this old Sun machine? I must say that I immediately liked it.

SunFire v100: Inside view

Taking out the HDD and replacing it with another drive was a real joy compared to what I had feared that I’d be in for. The drive bays are fixed using a metal clamp that snaps into a small plastic part (the lavender ones in the picture). I’ve removed the empty bay and leaned it against the case so that it’s easier to see what they look like. It belongs where the ribbon cable lies – rotated 90 degrees of course.

Old x86 server for comparison – getting two drives in there is very unpleasant to do…

All the other parts are easily accessible as well: The PDU in the upper left corner of the picture, the CDROM drive in the lower right, as well as the RAM modules in the lower left one. It’s all nicely laid out and well assembled. Hats off to Sun, they really knew what they were doing!

Lights out!

I briefly mentioned the LOM before. It’s short for Lights-Out Management. You might want to think IPMI here. While this LOM is specific to Sun, its basic idea is the same as the wide-spread x86 management system: It allows you do things to the machine even when it’s powered off. You can turn it on for example. Or change values stored in the NVRAM.

LOM starting up

How do we access it? Well, the machine has a RJ-45 socket for serial connections appropriately labeled “LOM”. The server came with two cables to use with it, one RJ-45 to DB26 (“parallel port”) used with e.g. a Sun Workstation, and a RJ-45 to DB9 (“serial port” a.k.a. “COM port”). Then you can use any of the various tools usually used for serial connections like cu, tip or even screen.

Just plug your cable into say your laptop and the other end into the A/LOM port, then you can then access the serial console. If you plug in the power cable of the SunFire machine now, you will see the LOM starting up. Notice that the actual server is still off. It’s in standby mode now but the LOM is independent of that.

LOM help text

By default, the LOM port operates in mixed mode, allowing to access both the LOM and the serial console. These two things can be separated if desired; then the A port is dedicated to the LOM only and the console can be accessed via the B port.

In case you have no idea how to work with the LOM, there’s a help command available to at least give you an idea what commands are supported. Most of these commands have names that make it pretty easy to guess what they do. Let’s try out some!

LOM monitoring overview (powered off)

Viewing the environment gives some important information about the system. Here it reveals that ALARM 3 is set. Alarm 1, 2 and 3 are software flags that don’t do anything by themselves. They can be set and used by software installed on the Solaris operating system that came with the machine.

I really have no idea why the alarm is set. It was that way when I got the server. Even though it’s harmless, let’s just clear it.

Disabling alarm, showing users and booting to the ok prompt

The LOM is pretty advanced in even supporting users and privileges. Up to four LOM users can be created, each with an individual password. There are four privileges that these can have: A for general LOM administration like setting variables, U for managing LOM users, C to allow console access as well as R for power-related commands (e.g. resetting the machine). When no users are configured, the LOM prompt is not protected and has full privileges.

OpenBoot prompt

It is also possible to set the boot mode in the LOM. By doing this, the boot process can e.g. be interrupted at the OpenBoot prompt which (for obvious reasons) is also called the ok prompt. In case you wonder why the command is “boot forth” – this is because of the programming language Forth which the loader is written in (and can be programmed in).

ok prompt help

In the ok prompt you can also get help if you are lost. As you can see, it is also somewhat complex and you can get more help on the respective areas that interest you.

Resetting defaults and probing devices

OpenBoot has various variables to control the boot sequence. Since I got a used machine, it’s probably a good idea to reset everything to the defaults.

From the ok prompt it’s also possible to probe for devices built into the server. In this case, an HDD and a CDROM drive were found which is correct.

Setting NVRAM variables, escaping to LOM, returning to the ok prompt and resetting the machine

The ok prompt allows for setting variables, too, of course. Here I create an alias for the CDROM drive to get rid of working with the long and complex device path. Don’t ask me about the details of the latter however. I found this alias on the net and it worked. I don’t know enough about Solaris’ device naming to explain it.

Next I set the boot order to CDROM first and then HDD. Just to show it off here, I switch back to the LOM – using #. (hash sign and dot character). That is the default LOM escape sequence, however it can be reconfigured if desired. In the LOM I use the date command to display how long the LOM has been running and then switch back to the ok prompt using break.

LOM monitoring overview while the machine is running

Finally I reset the machine, so that the normal startup process is initiated and an attempt at booting from the CDROM is being made. I threw in a FreeBSD CD and escaped to the FreeBSD bootloader (which was also written in Forth until it was replaced with a LUA-based one recently).

Showing the monitoring overview while the machine is actually running is much more interesting of course. Here we can see that all the devices still work fine which is great.

LOM log and date, returning to console and powering off

Finally I wanted to show the LOM log and returning to the console. The latter shows the OK prompt now. Mind the case here! It’s OK and not ok. Why? Because this is not the OpenBoot prompt from the SunFire but the prompt from the FreeBSD loader which is the second-stage loader in my case!

That’s it for the exploring this old machine’s capabilities and special features. I just go back to the LOM again and power down the server.

Conclusion

The SunFire v100 is a very old machine now and probably not that useful anymore (can you say: IDE drive?). Still it was an interesting adventure for me to figure out what the old Sun platform would have been like.

While I’m not entirely sure if this is useful knowledge (SPARC servers in the wild are more exotic then ever – and who knows what the platform has evolved into in almost 20 years!), I enjoy digging into Unix history. And Sun’s SPARC servers are most definitely an important mosaic in the big picture!

What’s next?

Reviewing this old box without installing something on there would feel very incomplete. For that reason I plan to do another article about installing a BSD and something Solaris-like on it.

FreeBSD jails (1/2): Introduction and frameworks

This is the first part of a followup to the 4.11 mini-series of posts that I published here on my blog ([1], [2], [3] & [4]). Those posts covered installing an old FreeBSD 4.11 system and using pkgsrc to update some parts of the system from versions released in the last millennium (!) to those of today (or rather from December 2016).

Now we’re going one step further: We’ll migrate a 4.11 system into a jail on a fresh FreeBSD 11.0 host system! This will be dealt with in the next post. In this part I’ll introduce the jail concept for people who are not familiar with it, discuss some jail management frameworks and do a bit of preparation work.

Motivation

Why to send that old system to a jail? We’ve already done a bit of work on a 4.11 system. But that next step tightens security much more since we have a recent kernel, can make use of a state-of-the-art firewall, etc. This is not actually my main concern however. Those remaining machines are running internal processes, so security is nice but not critical. What is critical and really frightens me, is the age of the hard drives that they use… When you do a df and see that the total capacity of a drive is way below 100 GB, you can guess that drives like that have not been sold for… quite some years now! And knowing that drives are rather perfidious little things, I’d rather get rid of those as soon as possible.

What makes things worse: This is 4.11. There’s no GEOM available and thus no gmirror or anything. Yes, there are backups. But I’d rather not have to mess with something like that just to restore a system that should have been decommissioned years ago!

What are FreeBSD jails?

If you’re new to FreeBSD, you may have heard about jails without knowing what they actually are. If you’ve got a Solaris background, think zones. If you’re a Linux user, think containers. FreeBSD people usually cannot understand today’s container hype for the very simple reason that this OS has had jails for ages now (they were first available on development versions at the very end of the 20th century!). You are probably familiar with docker, Linux’s cgroups and all those facilities. FreeBSD basically does the same thing: Provide much enhanced but still very light-weight chroot environment.

The system in that jail is isolated from the actual host operating system so that escaping it isn’t trivial and requires special (insecure) settings. In general jails are meant to be secure after all, aren’t they? You can also give jails their own virtual network stack, you can cap their resource usage, and so on. It’s a mature, stable and secure way to do containerization of applications – or whole operating system installations.

Since FreeBSD is pretty good at keeping compatibility with old releases (for the next major release they are discussing whether to keep compatibility shims that allow executing binaries from commercial UNIX releases of the 80’s!!), it’s very much possible to stuff a 4.11 userland into a jail running on a recent version of the operating system. And that’s what we’re going to look at after getting some of the basics out of the way.

Jail frameworks

Jails in FreeBSD can be created and managed manually. We’re talking *nix here, after all! However there are several jail management frameworks available that promise to make things even easier or at least more convenient. If you want to get started with jails, I suggest that you pick a framework that sounds good to you feature-wise and simply begin playing with it. The whole thing is not that complicated actually.

If you’re just getting started, don’t bother with sysutils/warden. It was a popular jail manager created for PC-BSD but it is considered obsolete now.

One true classic among the jail frameworks is sysutils/ezjail. It still does the job and a lot of people continue to use it simply because they are already familiar with it. It allows for things like thin-provisioned jails that use a “base jail” which holds a complete FreeBSD userland, and uses nullmount to make that accessible in other jails. This makes updating very quick and convenient. It also leaves the base system read-only which may be another nice security feature at the cost of decreased flexibility. On the plus side Ezjail is independent on filesystems and thus works on UFS, too.

Then there was sysutils/iocage, a modern approach that made use of ZFS properties to store certain values. For that reason it demands at least one zpool on the system to be usable. It had been implemented in shell code which made it easy to read for admins who don’t have a lot of coding experience. It has however been rewritten and the old shell version is declared obsolete. If you like the idea of having it written in shell, take a look at sysutils/iocell, which is a fork of the last shell version and is actively maintained.

The new version of iocage is implemented in Python (initially it was announced to be rewritten in Go but that decision was revoked). Another warning: sysutils/py-iocage is the wrong version, too! Do not use it. The rewrite initially supported Python 2.7 – which is what this port is all about – but newer versions dropped support for that and now require Python 3.6. So the actual port you should use is sysutils/py3-iocage! That version requirement makes sense but unfortunately it leads to iocage not being available from packages (as FreeBSD’s default version of Python 3 is still 3.3 currently). So you probably need to build it from ports.

There’s one more framework that I want to mention: sysutils/cbsd. It is an advanced system that not only does jails but also allows to manage bhyve (FreeBSD’s modern hypervisor). If you’re looking for something more comprehensive you may well give it a try.

Choosing my framework

When I first wanted to get into jails, I tried out ezjail. It worked well for me and I’d still use it on a system with no ZFS available. When I have ZFS, though, I liked the iocage approach. Cbsd looks quite good but I didn’t do too much with it because I require things to be scriptable (like using the framework through SaltStack states – there’s even a Salt Formula for iocage, but that’s for the old implementation) and that doesn’t seem to be what they focus on. So when the shell based iocage was deprecated, I moved to iocell and then I was torn to stick with that or to return to the new iocage. In the end I opted for at least giving the rewrite a chance. Since it does its thing quite nicely, I’ve adopted that and will use it here.

You can of course use any other framework if you prefer a different one. Or you could setup your jail by hand – if you’re thinking about that, definitely take a look at this article by M. W. Lucas whom I actually have to thank not only for his post but kind of for this one as well. The thought of jailing the old systems crossed my mind once or twice, but if he hadn’t written about it in the past, I don’t think I’d ever have tried it (no, I read his post back in the day when he published it; yes, I wanted to try this for over three years now before I finally found some time to give it a shot!).

Preparations

Let’s install iocage first. Like stated before, we will have to build it from ports. Assuming a system that does not have a ports tree installed, this is how you can do it (there are better (cleaner) ways to build packages from ports, I planned to do a post about that topic since December!) but this old method still works:

# portsnap fetch extract
# cd /usr/ports/sysutils/py3-iocage
# make config-recursive
# make install clean

The new iocage needs a UTF-8 locale set on the system or it won’t run. Unfortunately this is not the standard in FreeBSD (this is something that I really hope for in FreeBSD 12!). There are multiple ways to do this, my preferred one to set it in /etc/login.conf:

Look for default:\ – it is followed by a block of indented lines, the last one being :umask=022:. Add another backslash at the end to make the block continue and add two more lines to it:

:charset=UTF-8:\
:lang=en_US.UTF-8:

Login configuration is one of the few things where the actual values are in fact stored in a database. The changes that you made to the text file won’t take effect even if you log out and back in. First run another command to update the DB according to the changes in the text file:

cap_mkdb /etc/login.conf

Now log out and back in. Try echoing $LANG – that should print the UTF locale’s name. If it does, you are set.

What’s next?

The next post will demonstrate how to use iocage to manage jails and show how to jail a FreeBSD 4.11 system so it runs on a FreeBSD 11 host.

Updating FreeBSD 4.11 (4/4) – Reflecting radical resurrection

In the first post of this mini series I wrote about legacy systems and installing FreeBSD 4.11. The second one shows how to configure the fresh system for remote access, bootstrap Pkgsrc, install Subversion to checkout FreeBSD code and update the system to the stable branch. And part three mainly deals with upgrading OpenSSH and the compilers. This post details some more updates until we reach the final state that’s possible with such an old system (without resorting to extreme means).

Planting a new tree

So far we’ve built some packages from 2013 and before. Using a current pkgsrc tree won’t work – the various pkgsrc tools that our system has are too old. It might not be too big a step but we can use a tree from the second half of 2014. Of course the newer SSH that we built before is not currently in the path so we need to create a temporary symlink before we can use CVS again:

# ln -s /usr/local/temp/bin/ssh /usr/local/pkgsrc/bin/ssh
# rehash

# cd /usr/pkgsrc
# cvs -danoncvs@anoncvs.netbsd.org:/cvsroot get -rpkgsrc-2014Q3 -P pkgsrc
# mv pkgsrc 14
# rm /usr/local/pkgsrc/bin/ssh

What the system looks like package-wise at the beginning of part 4

Most of pkgsrc’s tools make use of NetBSD’s compatibility library. Unfortunately the version that comes with the new pkgsrc tree won’t build anymore on an OS as old as FreeBSD 4.11. Same thing for libfetch. But the newer tools will work with older versions of that libs, too. So let’s prepare those two – libfetch need’s some more love to build:

# cd /usr/pkgsrc/13/pkgtools/libnbcompat
# bmake

# cd /usr/pkgsrc/13/net/libfetch
# cp Makefile Makefile.bak
# sed '14i\\
CFLAGS=         -Wno-error' Makefile.bak > Makefile
# bmake

As a next step we’re going to do two updates. Yes, in theory we could use “bmake update” to update packages. We will not do that. The reason is that we needed to abuse pkgsrc quite a bit so far by mixing package versions from various trees. Since “bmake update” is a destructive command (it will happily uninstall programs as well as packages depending on them!) this can lead to all sort of fun things like unresolvable dependencies and such.

If you like pain, go ahead. I’ve been there and I can confirm that it does work for some packages. For a lot of them actually. But in those cases where it doesn’t, it tends to do so much damage that you’re better off starting over than trying to fix things… That’s why I’ll show you a safer method instead: Build a package and update via pkg_add! Also it really starts to show how old the system is that we’re trying to build rather new packages on. More and more of them require some trickery to persuade them to build – but hey, we’re doing a gross thing here, anyway. So there’s no real reason to complain!

# cd /usr/pkgsrc/14/pkgtools/pkg_install
# bmake extract
# rm -r work/libnbcompat/*
# rm -r work/libfetch/*
# cp -R /usr/pkgsrc/13/pkgtools/libnbcompat/work/libnbcompat-20120702/* /usr/pkgsrc/14/pkgtools/pkg_install/work/libnbcompat/
# cp -R /usr/pkgsrc/13/net/libfetch/work/libfetch-2.34/* /usr/pkgsrc/14/pkgtools/pkg_install/work/libfetch/
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/14/packages/All/pkg_install-20130902nb1.tgz

# cd /usr/pkgsrc/14/pkgtools/bootstrap-mk-files
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/14/packages/All/bootstrap-mk-files-20140516.tgz

We made it so far, now let’s make a daring move and just download the latest stable pkgsrc tree – released in January 2017:

# cd /usr/pkgsrc
# fetch http://cdn.netbsd.org/pub/pkgsrc/stable/pkgsrc-2016Q4.tar.bz2
# tar xvjf pkgsrc-2016Q4.tar.bz2
# rm pkgsrc-2016Q4.tar.bz2
# mv pkgsrc 16

Updating pkgsrc tools

Since mid 2014, pkgsrc makes use of a new package, cwrappers. During my test run I somehow managed to just get this package built. Despite taking notes I have no idea what I did to just make it work! It must have been something that looked like a dead end (which is why I didn’t include it in my notes) but somehow provided “getline”… I tried to get it working again for almost one whole Sunday but for the life of me couldn’t find out what I previously did… In the end I gave up and tried to find another solution. I found one but while it is way more complex it at least means that I got rid of that nasty blocker again:

# cd /usr/pkgsrc/16/pkgtools/cwrappers
# bmake extract
# rm -r work/libnbcompat/*
# cp -R /usr/pkgsrc/13/pkgtools/libnbcompat/work/libnbcompat-20120702/* /usr/pkgsrc/16/pkgtools/cwrappers/work/libnbcompat/
# cp work/cwrappers-20161125/mi_vector_hash.c work/cwrappers-20161125/mi_vector_hash.c.bak
# cp work/cwrappers-20161125/fixup-libtool.c work/cwrappers-20161125/fixup-libtool.c.bak
# sed 's/stdint.h/inttypes.h/' work/cwrappers-20161125/mi_vector_hash.c.bak > work/cwrappers-20161125/mi_vector_hash.c
# sed 's/stdint.h/inttypes.h/' work/cwrappers-20161125/fixup-libtool.c.bak > work/cwrappers-20161125/fixup-libtool.c
# cp /usr/pkgsrc/14/pkgtools/cwrappers/files/bin/getline.c /usr/pkgsrc/16/pkgtools/cwrappers/work/cwrappers-20161125/getline.c.bak
# sed 's/ssize_t/size_t/' work/cwrappers-20161125/getline.c.bak > work/cwrappers-20161125/getline.c
# cp work/cwrappers-20161125/common.h work/cwrappers-20161125/common.h.bak
# sed '107i\\
size_t  getline(char **, size_t *, FILE *);' work/cwrappers-20161125/common.h.bak > work/cwrappers-20161125/common.h
# cp work/cwrappers-20161125/Makefile work/cwrappers-20161125/Makefile.bak
# sed '14i\\
LIB_SRCS+=      getline.c' work/cwrappers-20161125/Makefile.bak > work/cwrappers-20161125/Makefile
# bmake install clean clean-depends

Phew! Fortunately the next few updates are straight forward:

# cd /usr/pkgsrc/16/pkgtools/bootstrap-mk-files
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/bootstrap-mk-files-20160908.tgz

# cd /usr/pkgsrc/16/devel/bmake
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/bmake-20150505.tgz

# cd /usr/pkgsrc/16/net/tnftp
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/tnftp-20151004nb1.tgz

Next is another one that requires some patching:

# cd /usr/pkgsrc/16/pkgtools/digest/
# bmake extract
# cp work/digest-20160304/sha3.h work/digest-20160304/sha3.h.bak
# cp work/digest-20160304/keccak.c work/digest-20160304/keccak.c.bak
# cp work/digest-20160304/keccak.h work/digest-20160304/keccak.h.bak
# cp work/digest-20160304/sha3.c work/digest-20160304/sha3.c.bak
# sed 's/stdint.h/inttypes.h/' work/digest-20160304/sha3.h.bak > work/digest-20160304/sha3.h
# sed 's/stdint.h/inttypes.h/' work/digest-20160304/keccak.c.bak > work/digest-20160304/keccak.c
# sed 's/stdint.h/inttypes.h/' work/digest-20160304/keccak.h.bak > work/digest-20160304/keccak.h
# sed 's/stdint.h/inttypes.h/' work/digest-20160304/sha3.c.bak > work/digest-20160304/sha3.c
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/digest-20160304.tgz

Updating installed packages

Let’s update gettext first as a lot of packages need that one; xz is one of the packages that is linked against the old one and since libintl received a soname bump, it needs to be rebuilt. Since we want to update it anyway that’s not too bad. But there are other packages that we cannot update which depend on the old lib. So we’ll have to create a symlink to satisfy their need, too:

# cd /usr/pkgsrc/16/devel/gettext-lib
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gettext-lib-0.19.8.1.tgz

# cd /usr/pkgsrc/16/archivers/xz
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/xz-5.2.2.tgz

# ln -s /usr/local/pkgsrc/lib/libintl.so.9 /usr/local/pkgsrc/lib/libintl.so.7
# cd /usr/pkgsrc/16/devel/gettext-tools
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gettext-tools-0.19.8.1.tgz

Next in line is some more typical build dependencies:

# cd /usr/pkgsrc/16/devel/libtool-base
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/libtool-base-2.4.2nb13.tgz

# cd /usr/pkgsrc/16/devel/m4
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/m4-1.4.17.tgz

# cd /usr/pkgsrc/16/devel/bison
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/bison-3.0.4nb3.tgz

Just a few more packages and we’ll have updated most packages that can be updated (a few like zip and nbpatch can’t):

# cd /usr/pkgsrc/16/shells/bash
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/bash-4.4.005.tgz

# cd /usr/pkgsrc/16/lang/perl5
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/perl-5.24.0.tgz

# cd /usr/pkgsrc/16/devel/autoconf
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/autoconf-2.69nb7.tgz

# cd /usr/pkgsrc/16/devel/gmake
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gmake-4.1nb3.tgz

Rebuilding the compiler

First we need to update the two math libraries (and create another symlink so we can go on compiling):

# cd /usr/pkgsrc/16/devel/gmp
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gmp-6.1.2.tgz
# ln -s /usr/local/pkgsrc/lib/libgmp.so.13 /usr/local/pkgsrc/lib/libgmp.so.11

# cd /usr/pkgsrc/16/math/mpfr
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/mpfr-3.1.5.tgz

This unfortunately breaks the compiler. But we can still resort to the old GCC3 to build GCC4 again, right? Right:

# cp /usr/pkgsrc/13/distfiles/gcc-4.4.7.tar.bz2 /usr/pkgsrc/16/distfiles
# cp /root/.cshrc /root/.cshrc.bak
# sed 's:pkgsrc/gcc44:temp/gcc34:' /root/.cshrc.bak > /root/.cshrc
# source /root/.cshrc
# cc -v

While we probably still don’t need Object-C or Java we could in fact build GCC with them this time. Java requires Python2.7 installed but that can actually be built from the 2014 tree! The problem is that building Java requires more RAM than is available on 32 bit machines and will for that reason fail. However Java is deactivated by default for GCC 4.4 in the 2016 tree. So let’s just get rid of our custom options, build the default package and set the correct path again:

# cp /usr/local/pkgsrc/etc/mk.conf /usr/local/pkgsrc/etc/mk.conf.bak
# sed '/PKG_OPTIONS.gcc44/d' /usr/local/pkgsrc/etc/mk.conf.bak > /usr/local/pkgsrc/etc/mk.conf
# cd /usr/pkgsrc/16/lang/gcc44
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gcc44-4.4.7nb7.tgz

Almost everything updated!

Now we only have to restore the correct path and then we have the GCC4 back (with a newer patch level):

# cp /root/.cshrc.bak /root/.cshrc
# source /root/.cshrc
# cc -v
gcc version 4.4.7 (GCC)

Modern OpenSSH

There’s one more package to build that needs a bit of care: Pkgconf. It’s a simpler replacement for the older pkg-config but it won’t work out of the box for us:

# cd /usr/pkgsrc/16/devel/pkgconf
# bmake extract
# cp work/pkgconf-1.0.1/libpkgconf/stdinc.h work/pkgconf-1.0.1/libpkgconf/stdinc.h.bak
# cp work/pkgconf-1.0.1/getopt_long.h work/pkgconf-1.0.1/getopt_long.h.bak
# sed 's/stdint.h/inttypes.h/' work/pkgconf-1.0.1/libpkgconf/stdinc.h.bak > work/pkgconf-1.0.1/libpkgconf/stdinc.h
# sed 's/stdint.h/inttypes.h/' work/pkgconf-1.0.1/getopt_long.h.bak > work/pkgconf-1.0.1/getopt_long.h
# bmake install clean clean-depends

Finally the time has come to do what I wanted to do in the first place, provide a recent version of OpenSSH! Of course it’s also necessary to generate new host keys once more. And then, just to prove everything works when the machine boots, let’s just restart the machine after adjusting the sshd path:

# cd /usr/pkgsrc/16/security/openssh
# bmake install clean clean-depends
# rehash
# ssh -V

# ssh-keygen -f /usr/local/pkgsrc/etc/ssh/ssh_host_rsa_key -N '' -t rsa
# ssh-keygen -f /usr/local/pkgsrc/etc/ssh/ssh_host_dsa_key -N '' -t dsa
# mkdir -p /usr/local/pkgsrc/run

# cp /etc/rc.conf /etc/rc.conf.bak
# sed 's:temp/sbin:pkgsrc/sbin:' /etc/rc.conf.bak > /etc/rc.conf
# shutdown -r now

Generating new host keys for OpenSSH

Now we need to remove the vierelf entry in ~/.ssh/known_hosts before we connect again. Doing so in verbose mode even shows that the 4.11 box now has a newer version of OpenSSH installed that the FreeBSD 11 workstation that I use to connect to it! 😀

FreeBSD 4.11 running a newer OpenSSH than my FreeBSD 11.0 workstation!

Conclusion

FreeBSD 4.11 is really, really, really old now. But you can get surprisingly far in running somewhat modern software on it – more recent software at least than I initially thought would be possible! And you? What was your bet? Would you have guessed that I’d make it up to the 2016Q4 pkgsrc tree and even install the latest version of OpenSSL and OpenSSH?

Here’s a little summary of some important program updates:

binutils 2.12.1 (2002) -> binutils 2.17 (2006)
perl 5.005 (1998) -> perl 5.8 (2002) -> perl 5.18 (2013) -> perl 5.24 (2016)
GCC 2.95.4 (2001) -> GCC 3.4.6 (2006) -> GCC 4.4.7 (2012)
OpenSSH 3.5 (2002) -> OpenSSH 4.6 (2007) -> OpenSSH 7.3 (2016)

Not too bad, eh? The notable exception here is binutils. Newer versions would probably be possible but there’s a gap in pkgsrc – which stuck with 2.17 for a long time and then directly moved to 2.22 which no longer builds on FreeBSD 4.11. GCC 4.5.3 does build BTW but something goes sideways and the comparison of stage 2 and 3 fails for quite some files.

I’ve met my initial goal to provide a newer version of OpenSSH, surpassing all expectations that I had. There’s room for more of course but that’s not worth another post. I’m going to add sudo and since Python 2.7 can be built it might even be possible to manage the 4.11 servers using salt-ssh (the ordinary SaltStack doesn’t work as it requires ZeroMQ which looks like it cannot be built)! We have a recent version of bash and can thus do some pretty nifty things with the right .bashrc.

This whole adventure took far longer than I had anticipated – a bit over a month instead of the intended two weekends! But that was mostly because I decided to start over with a clean system several times to ensure that everything works as I wrote it down here (and because GCC4 simply takes so long to build on the only spare machine that I had for this…). But it has been an interesting ride and I don’t regret spending some time on the legendary FreeBSD 4.11!

Oh, and my special thanks to everybody involved with Pkgsrc! I usually don’t have much use for NetBSD but Pkgsrc is extremely useful. I might use it in the future on other systems (like Linux), too. And thanks to you for reading. I hope that you enjoyed it as well!

Updating FreeBSD 4.11 (3/4) – Neophyte’s notorious necromancy

The first post of this mini series was about legacy systems in general and about what installing the old FreeBSD 4.11 is like. In the second one I showed the initial configuration of the system, how to SSH into it despite the obsolete DSA host key and how to bootstrap pkgsrc, NetBSD’s portable ports tree. I also covered the installation of SVN, checking out of the 4.11-STABELE code and updating the system. This post will cover installing newer software.

Any bets?

So far we have a pkgsrc tree from mid 2007 and things seem to be working. However that’s pretty close to 4.11’s release in 2005 and thus not too amazing. Working with such an old system there are plenty of cases which mean “game over”. Here are just three errors of that kind which you can encounter trying to build more modern software:

/usr/libexec/elf/ld: cannot find -lpthread

There’s no modern pthreads available on 4.11. Game over.

/usr/include/sys/resource.h:58: error: field 'ru_utime' has incomplete type

We’ll have to do with very old system headers missing a lot of what we take for granted today. Game over again.

fileio.o(.text+0x354): undefined reference to `towupper'
collect2: ld returned 1 exit status

Sorry, that ancient libc that we have on our system doesn’t provide that symbol. Game over yet again.

How far do you think can we take it in building and installing more recent software? Make a guess now and see if you were right! To be honest I was not expecting the end result. Not at all. So let’s get back to work!

In for a screening

We’re going to compile a lot of stuff this time – building SVN and dependencies was just a warm up. And what do you do when you’re building stuff remotely over SSH? You’re doing so in a screen or tmux session of course. Neither is part of the base system so we’ve got to build one. Tmux was not yet available in 2007 so it’s not too hard a choice:

# cd /usr/pkgsrc/07/misc/screen
# bmake install clean clean-depends
# rehash
# screen

GNU screen started up and ready

If you don’t know screen do some reading because you will want to start using it (or rather the superior tmux). It basically allows you to detach from a session and reconnect later – and your programs will continue running on the remote system even while you’re logged out. You can also resume the session from another terminal or computer, share sessions, etc. And that’s just one of the things that it does. There are other features like allowing you to have multiple shell instances in just one terminal between which you can switch back and forth (think tabs of a browser) and a lot more. Should you not like this (what’s wrong with you?!), fine. Don’t install screen. It’s optional.

Replacing the front door lock

Now it’s time to take care of the main problem of our system: That darned version 3.5 of OpenSSH! Let’s build whatever our pkgsrc tree has to offer:

# cd /usr/pkgsrc/07/security/openssh
# bmake install clean clean-depends
# rehash
# ssh -V
OpenSSH_4.6p1, OpenSSL 0.9.7d-p1 17 Mar 2004

Still far from a modern version of OpenSSH but also a lot better. And the best thing: It supports RSA keys. Let’s generate host keys with this newer SSH and make it the version that FreeBSD launches during startup:

# ssh-keygen -f /usr/local/temp/etc/ssh/ssh_host_key -N '' -t rsa1
# ssh-keygen -f /usr/local/temp/etc/ssh/ssh_host_rsa_key -N '' -t rsa
# ssh-keygen -f /usr/local/temp/etc/ssh/ssh_host_dsa_key -N '' -t dsa
# mkdir -p /usr/local/temp/run
# echo 'sshd_program="/usr/local/temp/sbin/sshd"' >> /etc/rc.conf

Ok, everything is in place. We could reboot now – or just kill off the old daemon and launch the new one. Let’s first look for SSHD and see which PID it has (this of course varies from system to system!):

# ps aux | grep sshd

Replacing SSHD

Got it? Great, let’s kill it (your SSH connection is maintained by a child and it’s generally save to kill the parent. You won’t lose your SSH connection!), start the new one and ensure that it’s running:

# kill [PID on your system]
# /usr/local/temp/sbin/sshd
# ps aux | grep sshd

What’s this? It looks like it’s not running! Yes, it looks like it but actually it should be running… Let’s grep again:

# ps aux | grep local

This does return one process – and trust me it’s actually our new sshd. What’s happening here is this: The output of ps is truncated because more wouldn’t fit on the screen. And only that data is handed to grep! So the process with the name /usr/local/temp that we found (see the screenshot above) is actually /usr/local/temp/sbin/sshd with the last part of it cut off… This is why grep doesn’t find “sshd”. There’s a funny way to fix this, though: Maximize your terminal emulator so that more space is available. Then grep will find sshd!

Now we can quite the old SSH session so we can make one with the new server. We can even keep our screen session open, but we need to detach from it by pressing CTRL-A and then D before we logout from vierelf:

[detached]

# logout
> exit
Connection to 192.168.1.5 closed.

Time to edit your known hosts and get rid of the former host key for vierelf or else you’ll see that scary SSH warning when you try to login again. Oh, and you can leave out that compatibility option from now on – which is a major step ahead! When you’re back in, you can resume the screen session:

% ssh kraileth@192.168.1.5
> su -
# screen -r

Connecting to the new SSH server (debug mode)

Compiler: from antediluvian to ancient

Alright. Currently we have the last version of the second generation of GCC on our system. We totally need to get our hands on something newer. How about updating the last version of generation three? Let’s try that! We only want the C and C++ compilers. Fortran is deactivated by default for this version (it would need GMP installed and the version of GMP that’s in the tree requires GCC3. It’s a good idea to avoid that potential circular dependency). However Java and Object-C are activated. There’s no need to waste time on them, they should be deactivated as well. The following sed command may look a bit complex, but it’s not that bad. Just copy all three lines that make up that single command and you’re good to go:

# cd /usr/pkgsrc/07/lang/gcc34
# cp Makefile Makefile.bak
# sed -e '64,65d' -e '63a\\
BUILD_JAVA?=    NO' -e '63a\\
BUILD_OBJC?=    NO' Makefile.bak > Makefile
# bmake install clean clean-depends

After installing that newer GCC, the path needs to be changed again so that the system picks it up instead of the older system compiler:

# vi /root/.cshrc

Prepend the following path to the PATH variable:

/usr/local/temp/gcc34/bin

Now let’s log out and in again and see if the new compiler is available:

# exit
[screen is terminating]
# logout
> su -
# screen
# cc -v
[...]
gcc version 3.4.6

Updating pkgsrc

Since we also have a more recent OpenSSH now, we can checkout a newer copy of pkgsrc from CVS! That takes a while, be patient. Even after it is finished downloading (and you see no new lines on the screen) it will still take some time to clean things up. This is normal and you have to wait a little longer. Don’t CTRL+C it as that would leave your tree in bad shape!

# cd /usr/pkgsrc
# cvs -danoncvs@anoncvs.netbsd.org:/cvsroot get -rpkgsrc-2009Q4 -P pkgsrc
# mv pkgsrc 09

Thanks to the newer SSH: CVS works now, too!

We’ll need some ports from there later. But since we have GCC 3 available now we can also grab an even newer copy and primarily use that one:

# cvs -danoncvs@anoncvs.netbsd.org:/cvsroot get -rpkgsrc-2013Q2 -P pkgsrc
# mv pkgsrc 13

We’re going to start a fresh environment, using only GCC (and sshd) from the old one. To do so we first bootstrap the pkgsrc from 2013 into a new directory:

# mkdir /usr/local/pkgsrc
# cd /usr/pkgsrc/13/bootstrap
# ./bootstrap --prefix=/usr/local/pkgsrc --varbase=/usr/local/pkgsrc

The next step is to adjust the path variable so that the binaries from the new location are being used. To do so we need to replace /usr/local/temp with /usr/local/pkgsrc for both sbin and bin. Don’t change the compiler path, though! GCC 3 will remain in temp. After logging out and back in, screen is no longer in PATH so we need to execute it with the absolute path:

# cp /root/.cshrc /root/.cshrc.bak
# sed -e 's:temp/bin:pkgsrc/bin:' -e 's:temp/sbin:pkgsrc/sbin:' /root/.cshrc.bak > /root/.cshrc
# exit
# logout
> su -
# /usr/local/temp/bin/screen

Cherry-picking dependencies

This gives us a way to easily build software from 2013. Let’s continue on by fetching some source tarballs by hand that are no longer available on the mirrors that pkgsrc knew for them:

# cd /usr/pkgsrc/09/distfiles
# fetch http://ftp.cc.uoc.gr/mirrors/NetBSD/packages/distfiles/binutils-2.17.tar.gz
# fetch http://ftp.cc.uoc.gr/mirrors/NetBSD/packages/distfiles/pkg-config-0.23.tar.gz

The following part is not too interesting: We’re going to build the dependencies in preparation for the next big step. In general we try to build the newest version possible (2013) but resort to old (2009) or even older (2007) where necessary if newer versions don’t build for various reasons:

# cd /usr/pkgsrc/13/converters/libiconv
# bmake install clean clean-depends

Zip from 2009 and onwards is incompatible with FreeBSD 4.11’s libc. And the 2007 version expects tar in a location where there’s none on our system. Instead of building tar we can safely symlink it:

# ln -s /usr/bin/tar /usr/local/pkgsrc/bin/tar
# cd /usr/pkgsrc/07/archivers/zip
# bmake install clean clean-depends

The binutils are a special case. The port normally builds the programs of which it consists with a prefix so they don’t get in the way of the system binaries. Since we actually want to use them instead of the old stuff from the base system, we need to get rid of that prefix:

# cd /usr/pkgsrc/09/devel/binutils
# bmake GNU_PROGRAM_PREFIX='' install clean clean-depends
# rehash
# ld -v
GNU ld version 2.17

The next few are trivial:

# cd /usr/pkgsrc/09/devel/gettext-tools
# bmake install clean clean-depends

# cd /usr/pkgsrc/13/devel/m4
# bmake install clean clean-depends

# cd /usr/pkgsrc/09/devel/bison
# bmake install clean clean-depends

The bash port from 2013 would draw in a newer version of gettext which would not build. But bash can actually be built with the old one, too. So we have to make a simple change in the buildlink file for gettext in 2013’s pkgsrc tree:

# cd /usr/pkgsrc/13/devel/gettext-lib
# cp buildlink3.mk buildlink3.mk.bak
# sed 's/0.18/0.14/g' buildlink3.mk.bak > buildlink3.mk

With that change the next port can be built:

# cd /usr/pkgsrc/13/shells/bash
# bmake install clean clean-depends

Next in line is perl. The 2013 port would however build with dtrace support by default – which was of course not available on 4.11. Therefore it needs to be switched off by making an addition to the pkgsrc config file:

# vi /usr/local/pkgsrc/etc/mk.conf

Add the following line at the end of the file (but above .endif):

PKG_OPTIONS.perl=       -dtrace

Now let’s build the last few dependencies:

# cd /usr/pkgsrc/13/lang/perl5
# bmake install clean clean-depends

# cd /usr/pkgsrc/13/archivers/xz
# bmake install clean clean-depends

# cd /usr/pkgsrc/09/devel/autoconf
# bmake install clean clean-depends

Compiler: from ancient to old

With this all dependencies from earlier than 2013 in place we are good to go for the biggest update. We’re still not interested in Java and Object-C, so let’s edit pkgsrc’s configuration again:

# vi /usr/local/pkgsrc/etc/mk.conf

and add one more line (e.g. after the perl one):

PKG_OPTIONS.gcc44=      -gcc-java -gcc-objc

Building the newer version of GCC means building two more dependencies as well, one of which is libgmp. GMP is the first package so far that uses C++ and in fact our C++ compiler has been broken the whole time. Luckily a symlink can heal it and another one will make GCC happy so that we can finally build it – which takes quite a bit of time (I’ve seen the compilation stop at one point and I’m not sure what happens there. But just calling bmake again will eventually complete the build process!):

# ln -s /usr/local/pkgsrc/lib/libiconv.so.7 /usr/lib/libiconv.so.7
# ln -s /usr/local/temp/gcc34/lib/libgcc_s.so.1 /usr/lib/libgcc_s.so.1
# cd /usr/pkgsrc/13/lang/gcc44/
# bmake install clean clean-depends

Once it’s build, we need to change our PATH so that the newer GCC is the primary compiler:

# mv /root/.cshrc /root/.cshrc.bak
# sed 's:temp/gcc34:pkgsrc/gcc44:' /root/.cshrc.bak > /root/.cshrc

Now all that we have to do is log out and back in:

# exit
# logout
> su -
# /usr/local/temp/bin/screen

Let’s take a look if the new compiler responds to cc (and fix c++ support along the way):

# ln -sf /usr/local/pkgsrc/gcc44/lib/libgcc_s.so.1 /usr/lib/libgcc_s.so.1
# cc -v
[...]
gcc version 4.4.7 (GCC)

GCC 4.4.7 running on FreeBSD 4.11

Yes, we really have GCC 4.4 running on FreeBSD 4.11! While it’s certainly not a modern compiler, it’s recent enough to build a lot of software. The latest release of OpenBSD, version 6.0 released on September 2016, still comes with GCC 4.2, BTW! Yes, OpenBSD maintained that all the time and heavily patch it. Still we now actually have a compiler available on FreeBSD 4.11 from 2005 which is two major versions newer!

With this we’re kind of back in business. But this post is already becoming quite long and for that reason I’m putting the “grand finale” off to one more post. See you there for the final outcome of this “little” experiment (which I hadn’t intended to write more than three posts for, but there you have it).

Updating FreeBSD 4.11 (2/4) – Digging up old graves

In part one I wrote about Legacy systems in general and showed a FreeBSD 4.11 installation for those of my readers who are interested in software history.

This post is about the first part of updating this fresh 4.11 system to a state that’s a bit less catastrophic. Remember: FreeBSD 4.11 was released in 2005 – however the ABI of each release is carved in stone with a .0 release. Which means that the software in the base system is from 4.0 and thus we venture back into the last millenium: 1999!

Initial state

To give you an idea what this means, here are a few program versions:

Various program’s versions in 4.11’s base system

So we have these programs among others:

GCC 2.95.4
Binutils 2.12.1
Perl 5.0
OpenSSH 3.5

To make matters worse, the ports tree for FreeBSD 4.11 is pretty dead, too. It’s important to get newer compilers running, but around 2005 FreeBSD used special releases to build GCC from (“gcc-core”) and I was not able to find a single mirror on the net that still holds those old and exotic files! Out of luck here. We’ll have to do without those ports.

“Modernizing” this is going to be interesting… Considering how fast the IT world moves, all of this is just as dead as it gets. So let’s prepare for the (code) smell and start digging up an old grave, shall we?

Allowing remote connection

After a passwordless login as root it makes sense to set up the right keymap (if you don’t use the default, that is). I had no idea how to do it on 4.11 and so I just gave the usual way of doing it a try – and was met with success:

# kbdmap

Looks like the keymap selection has not changed in all the time! Let’s try to make it persistent:

# echo keymap=german.iso.kbd >> /etc/rc.conf

I tried it out and it did just what I wanted. Time to try to add a regular user:

# adduser

The script is a bit different from what we’re used to today but in the end it does the same thing: Allow us to create a user. It’s important to add this user to the wheel group so that we’re able to su to root. However we need to give root a password first:

The useradd script in FreeBSD 4.11

# passwd

Ok, sshd should be running. Let’s check just to be sure:

# ps aux | grep sshd
[...]
root        75  0.0  0.1  2600 2040  ??  Is    6:26AM   0:00.11 /usr/sbin/sshd

Looks good. What about connectivity? Let’s see:

# ifconfig
vr0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        ether 00:05:5d:96:fa:f9
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
[...]

Nope, no connection. Luckily I paid attention when I the installer started from the CD and somewhere the strange-looking path of /stand caught my eye. Doing a little research, I found out that this directory used to be part of the OS and was removed in early FreeBSD 5 as it was mostly redundant with /rescue. What was in there, you ask? Have a look yourself:

# ls /stand
-sh             etc             minigzip        rm              tunefs
[               find            mount_mfs       route           usbd
arp             fsck            mount_nfs       rtsol           usbdevs
boot_crunch     gunzip          newfs           sed             zcat
camcontrol      gzip            pccardc         sh
cpio            help            pccardd         slattach
dhclient        hostname        ppp             sysinstall
dhclient-script ifconfig        pwd             test

Network configuration

There’s our friend sysinstall. I already said that it does more than just install the system. So let’s bring it up now:

# /stand/sysinstall

Sysinstall to configure the network

There we choose Networking -> Interfaces -> the appropriate NIC. No, I don’t want IPv6 and yes, DHCP is the thing.

Interface configuration using sysinstall

I’ve called the system vierelf which would be “foureleven” in English because I couldn’t think of anything better. And it’s just a test system anyway. Does the connection work now?

# ping elderlinux.org
PING elderlinux.org (212.77.232.71): 56 data bytes
64 bytes from 212.77.232.71: icmp_seq=0 ttl=54 time=24.993 ms

A little housekeeping

Alright! Let’s just take a look what services are listening right now:

# sockstat -4 -l
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      
root     dhclient   281    7 udp4   *:68                  *:*                  
root     sendmail    84    4 tcp4   *:25                  *:*                  
root     sendmail    84    6 tcp4   *:587                 *:*                  
root     sshd        75    4 tcp4   *:22                  *:*                  
root     syslogd     61    5 udp4   *:514                 *:*

Ugh! Time to make a few changes in /etc/rc.conf to deactivate any daemons except for SSH (which we need). This may not be strictly necessary but we want to improve the security of this system, right? And I wouldn’t trust those crusty old daemons at all. And whatever is not running won’t cause us any problems. So let’s get rid of them with extreme prejudice!

# vi /etc/rc.conf

To do so, add daemonname_enable=”NO” for each daemon and in case of sendmail use sendmail_enable=”NONE”.

FreeBSD 4.11 rc.conf with most daemons disabled

If you reboot now, sendmail and syslogd as well as cron, usbd and inetd will be disabled. That’s a starting point in securing the system. Let’s move on.

Replacing a dead tree

I’ll connect to the 4.11 box remotely over SSH because it’s much more convenient to have my trusty terminal at hand and to be able to copy and paste stuff:

% ssh kraileth@192.168.1.5
Unable to negotiate with 192.168.1.5 port 22: no matching host key type found. Their offer: ssh-dss

Ouch! I cannot even SSH into the system because its version of OpenSSH is so old that it only offers ssh-dss keys which have been deprecated for quite a while and disabled by default in OpenSSH >=7.0. So to connect to that old server, I have to tell my SSH client to accept ssh-dss for this connection:

% ssh kraileth@192.168.1.5 -oHostKeyAlgorithms=+ssh-dss

SSH login using “-oHostKeyAlgorithms=+ssh-dss”

Ok, I’m in. But what now? We cannot use FreeBSD’s ports tree but I strongly prefer some means of package management over installing stuff using make install. So how do we accomplish this? Enter pkgsrc. Pkgsrc is basically NetBSD’s fork of the FreeBSD ports tree. Being a NetBSD project however, it’s not limited to just NetBSD. It’s a truely portable way of building and managing software (I might write a separate post about it some time).

There’s just one problem: Downloading and decompressing the latest pkgsrc release (currently 2016Q4) won’t complete the bootstrapping process. Obviously FreeBSD 4.11 is no longer supported – which is not so much of a surprise. Time to try out older releases! After doing so I found out that 2009Q4 seems to be the last release to bootstrap successfully.

But here’s another problem: Pkgsrc doesn’t seem to keep older releases around and I also haven’t found them mirrored anywhere on the net. Pkgsrc uses CVS, however. So it’s possible to checkout older versions. FreeBSD comes with CVS as part of the base system. Unfortunately CVS works over SSH. And NetBSD’s CVS server won’t accept ssh-dss (which totally makes sense)! Since we don’t control the server, there’s also no way to just add a parameter or something to make it work. It simply doesn’t work that way.

Time to get CVS on my slightly more modern FreeBSD 11, do the checkout there and tar it all up to copy it over via scp! We’re going to get 2007Q2 instead, though, since we need things that won’t work on FreeBSD 4.11 with later versions. Oh, and if you’re not familiar with CVS, don’t worry. You don’t need to know what modules or tags are. Just copy the commands that I prepared for you and you’re good to go:

% sudo pkg install cvs
% cvs -danoncvs@anoncvs.netbsd.org:/cvsroot get -rpkgsrc-2007Q2 -P pkgsrc
% tar cvjf pkgsrc2007Q2.tbz2 pkgsrc/*
% scp pkgsrc2007Q2.tbz2 kraileth@192.168.1.5:/usr/home/kraileth

Then back on vierelf the next step is to prepare some directories and extract the pkgsrc tarball:

# mkdir -p /usr/local/temp /usr/pkgsrc
# cd /usr/pkgsrc
# mv /usr/home/kraileth/pkgsrc2007Q2.tbz2 .
# tar xvjf pkgsrc2007Q2
# rm pkgsrc2007Q2.tbz2
# mv pkgsrc 07

Bridgehead in hostile territory

Now we can bootstrap pkgsrc:

# cd /usr/pkgsrc/07/bootstrap
# ./bootstrap --prefix=/usr/local/temp --varbase=/usr/local/temp --pkgdbdir=/usr/local/temp/db

Pkgsrc 2007Q2 bootstrap complete!

Pkgsrc has been bootstrapped successfully! We just need to adjust the path variable so that the system picks up binaries from the new paths (and make those take precedence over the old system binaries). We could just change the PATH variable but it’s better to make the changes persistent. So let’s add the two new paths in the shell’s rc file in front of the others:

# vi /root/.cshrc

This is what needs to be prepended:
/usr/local/temp/sbin /usr/local/temp/bin

Root’s .cshrc file for the first phase pkgsrc

Now simply log out and become root again to have the new environment:

# logout
> su -

As a last step check if everything is right and we can access binaries in both paths:

# which bmake
/usr/local/temp/bin/bmake
# pkg_info 
bootstrap-mk-files-20061111 *.mk files for the bootstrap bmake utility
bmake-20051105nb3   Portable (autoconf) version of NetBSD 'make' utility
tnftp-20050625nb1   The enhanced FTP client in NetBSD
mtree-20070710      Utility for mapping and checking directory hierarchies
pax-20060202nb1     POSIX standard archiver with many extensions
pkg_install-20070710 Package management and administration tools for pkgsrc

Excellent! Now we have a working replacement for FreeBSD’s dead ports tree. This is definitely something that we can build upon.

Bring on some stability!

Lame pun, I know. Nevertheless it makes sense to… err… update the OS to the latest version. This is what we currently have:

# uname -a
FreeBSD vierelf.localdomain 4.11-RELEASE FreeBSD 4.11-RELEASE #0: Fri Jan 21 17:21:22 GMT 2005     root@perseus.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386

We’re on 4.11-RELEASE. The latest code for each release cycle is always in the -STABLE branch. Believe it or not: The latest code change was in April of 2014! The traditional way of getting FreeBSD code was over CVS. No, this time the problem is not ssh-dss. FreeBSD migrated from CVS to SVN (subversion) in 2008. FreeBSD CVS servers have been removed years ago. Therefore the old tools like cvsup are useless. Subversion is needed to checkout the source.

Luckily we have our pkgsrc ready. This old release has a very old port for subversion but that’s fair enough. There are a few source tarballs that are no longer available from the mirrors that pkgsrc knew for them. Not a big problem, we can download those manually:

# cd /usr/pkgsrc/07/distfiles
# fetch http://archive.apache.org/dist/httpd/httpd-2.0.61.tar.bz2
# fetch http://repository.timesys.com/buildsources/e/expat/expat-2.0.1/expat-2.0.1.tar.gz
# fetch http://download.nust.na/pub2/openpkg1/sources/DST/pkgconfig/pkg-config-0.21.tar.gz

Now we can build subversion:

# cd /usr/pkgsrc/07/devel/subversion-base
# bmake install clean clean-depends

Various dependencies will be downloaded, built and installed. Eventually subversion will be installed and available on the system. Time to tell the shell to look for new binaries and then checkout the stable source code:

# rehash
# svn co svn://svn.freebsd.org/base/stable/4 /usr/src

SVN checkout of the 4.11-STABLE code

The FreeBSD 4 base system never knew anything beyond CVS and cannot cope with the .svn directories that the svn checkout creates. World builds but the installation fails like this:

install: /usr/libdata/perl/5.00503/./.svn/text-base/Cwd.pm.svn-base: No such file or directory

Therefore it’s necessary to get rid of those disruptive directories:

# cd /usr/src
# find . -iname '.svn' -exec rm -rf {} \;

Now we can build world and kernel, install both and reboot the system:

# make buildworld
# make buildkernel
# make installkernel
# make installworld
# shutdown -r now

When the system comes back up we can SSH into it again. And there we can see that we’re on -STABLE now!

The newly built FreeBSD 4.11-STABLE

# uname -a
FreeBSD vierelf.localdomain 4.11-STABLE FreeBSD 4.11-STABLE #0: Sat Jan 28 11:53:06 GMT 2017     root@vierelf.localdomain:/usr/obj/usr/src/sys/GENERIC  i386

That’s it for today. We’re not quite there yet, but we’ve laid the groundwork for many more updates to come. Those will be described in the coming two posts of this mini series.

Updating FreeBSD 4.11 (1/4) – Blast from the past

Ah, weekend. And it’s a nice day, too: The sun shines on the snowy landscape! A perfect day to go out with the children – or to embark on a vastly different kind of adventure… Yes, you read that title right. This is not about four FreeBSD 11 machines. It’s about one FreeBSD 4.11 machine in 2017!

Legacy systems in general

Why even bother with FreeBSD 4.11 today? Well, most people would probably respect historical interest and in fact I was keen on tinkering with a legendary old 4.x release and see if you can have some fun with it (spoiler: Oh yes, you can!). But this would be a task for times when I have absolutely nothing else to do. So it’s fairly obvious that there’s another reason.

“Old habits die hard” they say and there’s definitely truth to it. Another truth, however, is that in the IT world things often don’t go away easily. There’s still a considerable number of machines out there that run DOS. In production. Doing important stuff. Sometimes just because it works sufficiently well. But often they have special tasks that for some reason or another cannot be migrated as nobody has any idea clue how you’d do this.

You want examples? Sure. The first example is “Game of Thrones” author George R.R. Martin writing his books using WordStar 4.0 on a DOS machine. A lot of companies use DOS, too, but you usually don’t hear about that. And less than a month ago a new version of FreeDOS (v. 1.2) was released.

Or do you remember the French airport that was having trouble due to using Windows 3.11? That was in 2015 and they announced that they were planning to migrate their system officially by 2017 (but expectedly by 2019 or even 2021)!

FreeBSD 4.11

One such case of a formerly very popular operating system that just won’t go away is FreeBSD 4.11. From the version number alone you can see that this is a very special one: Usually FreeBSD has three to four point releases in a release’s life cycle. FreeBSD 4 obviously was different. The main reason for that was that version 5 kind of was to FreeBSD what version 4 was for MS-DOS: Innovative in its day but disappointing especially in terms of stability (and performance) among other things.

I have to take up the cudgels for FreeBSD 5, though. It blazed a trail for important technology that we rely on today like the GEOM framework, porting the system to amd64, and so on. Implementing things like that meant digging deep into the system and doing pretty invasive changes. It totally paid off in the long run but back in the day people tended to avoid the early 5.x releases or the whole release cycle altogether.

There are many reasons why some companies still have FreeBSD 4.11 running: Critical systems that cannot be migrated, installed programs to which the source code was lost, etc. Things like that. Most of these systems are probably used internally only but I’m pretty sure that there’s also quite some 4.11 machines still serving data on the net…

Now what others do (and what their reasons for it might be) are not my primary concern – I can’t do anything about it anyway. However the company that I work for has its share of legacy systems, too. Especially younger admins hate them. Actually they are impressively stable and rarely if ever have problems. In fact they keep running, silently doing their job. But now and then you have to do some changes to them and that’s where the trouble starts: They simply cannot be compared to the modern-day Linux systems that we’re used to administer daily!

Preparations

I kept wondering if that situation could be improved. It wouldn’t be worth the effort to do this during work time but I was curious enough about the legendary 4.11 release to give it a try. There’d surely be enough to learn. And should I succeed in bringing a test system into a more modern state there would also be a benefit for us at work.

We still have all those old CDs at work but I made my decision impulsively on Saturday. Fortunately ISOs of the old Walnut Creek CD-ROMs are available for free download. Thanks, Archive.org! I burned it to a CD and was able to boot from it.

I actually installed the OS on real hardware. First on an old HP compaq nx9010 laptop that I got for free a while ago. Things worked pretty well, but since what I’m trying to do is almost all about compiling software, the compilation times proved to be a bit long. Therefore I decided to redo the previous steps on faster hardware – and since amd64 PCs are capable of running i386 programs, all was well. I just had to disable AHCI in the BIOS since FreeBSD 4.11 doesn’t know what to do with that.

The actual installation was interesting enough (for people like me who enjoy history) and therefore I decided to redo it again in VirtualBox to take screenshots and describe the installation process for the remainder of this post.

Installation

Quite some time has passed since FreeBSD 4.11 was released in 2005. More than a decade is a lot in a man’s life but it’s a hell of a lot when it comes to software and operating system development! I had never before installed any version of FreeBSD older than 7.4 and even thinking about the differences between 7.x and 11, many things changed. But what would 4.11 look like? Would I be able to install it without difficulties (I tried a very old version of OpenBSD once and quickly gave up because I didn’t really like the idea of manual partitioning of the drives…)?

FreeBSD 4.11’s Kernel Configuration Menu

I was surprised to run into something that I had never seen even before the installer started! There’s this Kernel Configuration Menu where you can choose to either boot the standard kernel or to customize it. Of course I was curious and so I decided to take a look at it.

FreeBSD 4.11 kernel drivers selection

This brings up a friendly little tool that allows for easy navigation through the driver tree. The screen is basically split into two parts: Active drivers and inactive drivers. Each one consists of several categories which are collapsed and can be expanded for a list of drivers.

Browsing FreeBSD 4.11’s additional network drivers

The standard kernel worked on both of my machines and so I didn’t play around with this after exploring it briefly. When the kernel had booted, I found myself in the good old Sysinstall utility. Yes, it’s pretty ugly and it looks a bit complicated – which is due to the fact that despite it’s name it does a lot more than just handle the installation. Yet in fact it’s really easy to navigate through.

I like the Express option that lets you perform an installation as quickly as possible by asking you only the absolute minimum of necessary questions. This goes as far as not asking for a root password and leaving it blank on your first login! Fair enough, it’s easy to set one yourself. Current versions of FreeBSD do not have this option. That may be due to the fact that it’s not terribly helpful for setting up a serious system. And the use case that I have for it here is definitely more of an exception!

FreeBSD 4.11’s Sysinstall: Main Menu

The first thing that’s essential is of course disk partitioning. If you don’t require any special setup, you can simply use the whole disk. I was glad to see that this can be done by simply pressing “A”!

Installing FreeBSD 4.11: Disk partitioning

Just like today FreeBSD 4.11 allows you to choose if you want to install a boot manager, just write the boot loader or don’t do anything. The first option can be used for dual-boot systems where you want to load either FreeBSD or another OS. The last option won’t work by itself; if you choose that, you need to work with an external boot manager that supports FreeBSD. In my case just installing the standard loader makes most sense.

Installing FreeBSD 4.11: Choosing boot options

Then there’s creating the disklabels. This is another thing where I’m happy to find an auto defaults option (simply pressing “A” again) because I just wanted a quick test system anyways and didn’t want to think too much about the disklabel layout!

Installing FreeBSD 4.11: Editing disk labels

Finally we have to select what to install. To make things easy, sysinstall offers several distribution sets to choose from. Minimal is all that I need for my experiments, but back in the day a lot of the other options were definitely quite handy. Two things stand out: You can choose to install FreeBSD + X11 for example. Current versions of FreeBSD don’t let you do that. You’ll always get a console only system and have to install X11 yourself (which IMO is something that leaves the barrier unnecessarily high for new users who don’t know FreeBSD nor its packaging system, yet!).

Sysinstall also allows for custom distribution selection. This gives you access menu that allows for really fine-grained choices. Again, FreeBSD’s installer cannot do that today. While it’s probably nice to have that option, it’s not a must-have. People who have use cases for that will probably know what they do and install FreeBSD manually, anyway. Still I like having it in the installer!

Installing FreeBSD 4.11: Choosing distribution sets

The only thing left is selecting what medium to install from – the CD in my case. Formatting a large drive takes quite a moment because we’re talking UFS1 here! Don’t start looking for a way to use UFS2, it’s not available in FreeBSD 4. The actual “minimal” installation is quite quick and just a moment later I can reboot and the system comes up just like you’d expect it to. Now that was the easy part of my adventure. Next stop: Brushing the dust away and updating that beast(ie)!