Cross-platform package building: Pkgsrc vs. Ravenports (2/2)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/cross-platform_package_building_pt2.gmi

The previous article (part 1) on cross-platform package management / package building covered the basics by taking a look at what some of the many difficulties are. It briefly described some common strategies, too.

Part 2 discusses two package building frameworks that were developed to allow for a cross-platform solution. It contains a short introduction of both, a somewhat detailed elaboration of how they compare as well as information on the test scenario and of course the results. It ends with a conclusion.

Contender 1: What is Pkgsrc?

Pkgsrc, pronounced package source, is NetBSD’s ports system. Since one of NetBSD’s primary goals is portability, it’s not much of a surprise that Pkgsrc is also portable across several platforms and architectures. It was originally derived from FreeBSD’s ports system but has later been redesigned in a complete rewrite. In a nutshell, you can think of a “port” as a buildsheet for some kind of software. You tell the system that you want it to build PostgreSQL, Firefox or whatever and if there is a port for that, the system knows what to do to accomplish that task.

The individual ports can be used directly by building the software on the host machine or to build the binary packages that can then be installed using a package manager. To build a package like Firefox, a lot of dependencies are required. If they are not already present on the system, they will be automatically built before it (there are ports for those programs or libraries as well).

If you want to know more about this, here are two articles that I wrote a couple of years ago: FreeBSD – building software from ports (part 1) and (part 2). They provide an introduction to the FreeBSD ports system. If you are unfamiliar with the concept, you probably want to skim over the first one and read it and the second one completely if you find it interesting.

Once you understand what FreeBSD’s ports tree is, you’ve got an idea of what Pkgsrc is, too: From the user’s perspective it has the same basic purpose.

Contender 2: What is Ravenports?

Ravenports is a much more recent take on the topic of package building. It was conceived by John Marino who had before contributed to the maintenance of thousands of ports in Pkgsrc and FreeBSD ports and who also was the inventor of dports (DragonFly BSD’s adaption of FreeBSD’s ports). After his attempts to modernize the old ports frameworks had failed, he decided to design a new one based on the lessons learned.

RP is meant to overcome limitations due to design decisions that looked reasonable more than two decades ago but have since proven to be obstacles. Modern-day tooling, high concurrency and a very high level of automation as well as convenient helpers for port maintenance is what it offers over others. It’s these features that allow it to reduce the army of porters usually required to maintain several thousand packages to a couple of capable people at most.

It is different from a classical ports system in it not being capable of building and installing software directly on the host system. It always uses a pristine “build jail” for each package in which only the relevant dependencies are installed before the actual build process begins. When the build is completed and the software packaged up, the build jail is torn down. FreeBSD ports and Pkgsrc allow for doing the same by using additional tools. With Ravenports this is the only supported (and built-in) mode of operation: Building packages properly, not installing software directly.

How do they compare?

I’ve used both Pkgsrc and Ravenports for private projects as well as for customers and thus benefited from them. While I like both, I won’t hide that I was so impressed by Ravenports that after trying it out I soon became involved in the project. So one could say that I’m biased and you cannot expect a fair comparison. I tried hard to come up with sensible criteria to put both to the test and I’ve asked other people (like the other two Advance!BSD members who are neutral) for their judgement.

I also deliberately treated Ravenports harshly during the comparison; there’s no bonus points from me. On the contrary: I proposed to evaluate Pkgsrc first, because I was under the impression that it would probably fit our project better (being already available for so many platforms and being very mature). It was only when we hit unexpected problems that I decided to go for a somewhat comprehensive evaluation and comparison.

A direct comparison of both candidates is actually pretty difficult to do. You could compare the platforms that both support. Pkgsrc wins this easily. However when it comes to support for bulk builds (building packages from all ports or a considerable subset), that’s broken on almost all of those platforms that Pkgsrc supports! And so when it comes to this kind of “fully supported” platforms, Ravenports takes it home.

One of the most obvious things you could compare is package count. Again this is not that simple because you’re comparing apples to oranges. Pkgsrc offers so many more packages than Ravenports; but among those are e.g. over 1,400 packages related to TeX – which Ravenports packages differently, ending up with only 15 packages for basically the same thing! Pkgsrc keeps various older compilers and versions of popular interpreters (Python, Ruby) in the tree even when they are no longer supported. Ravenports has a much stricter policy of trying hard to purge ports of software that has officially reached the status of EOL (End Of Life). Neither approach is inherently better than the other – it’s a question of what your preferences or requirements are.

On the other hand the number of Pkgsrc packages buildable for the various platforms (as listed below) is the minimum count (hence >=). The reason being that when not bulk-building, all dependencies for packages must be installed on the live system (and are not automatically deinstalled). Some dependencies will block others from being installed. And thus a considerable number of packages will fail to build as a result of it. So while I included the figures here to give an impression on how well each platform is supported, they have no absolute meaning and thus score neutral (total count is among the criteria in the “general” section, though).

So keep in mind that there are multiple problems that do not allow for simply comparing numbers. Think about what all the results mean and what is better for your use case. For example in our case a platform that does not support bulk builds is close to being worthless. However even partial support means that fixing bulk builds would be much less effort than having to bring it up on an entirely new platform.

Test conditions and environment

To make the already difficult tests at least compare as best as possible, a fresh VM was created for each system to test. Resources for those VMs were identical of course: 48 GB of RAM, 16 CPU cores (of a 3.0 GHz E5-2623 server) and 1 TB of disk space.

The tests started immediately after pkgsrc-2021-Q3 was published in late September and the Ravenports comparison is against the at that time latest released package set. So in this comparison newer additions (namely NetBSD support on Ravenports) are mentioned but treated absent for the score.

We are mostly interested in the *BSD operating systems and thus our focus is on those. As it might be of interest to a lot of people, we’ve included results for Linux and illumos, too, but gave the respective results a neutral score. The operating systems used in the tests were:

  • DragonFly BSD 6.0
  • FreeBSD 13.0
  • NetBSD 9.2
  • OpenBSD 7.0
  • Rocky Linux 8.4 (Community effort to replace what was CentOS before IBM… “repurposed” it)
  • OmniOSce r151038

For all criteria tested, a score is awarded. It can be very positive (++), positive (+), neutral (0), negative (-) or very negative (–). All of those added up lead to the final score.

Testing procedure

For all tests freshly installed operating systems were used. The installations were done as close to a standard / minimal installation as possible. This is true for DragonFly BSD and FreeBSD.

On NetBSD and OpenBSD all of the package sets were installed, since various X11 ports in Pkgsrc depend on those being present and would fail otherwise. Compared to many Linux distributions this is still a fairly slim installation, but it’s not minimal in the true sense of the word and thus should be noted here. On OpenBSD the standard partitioning scheme was also replaced with a simplistic one.

Since Linux does not know the concept of a base system, there’s also no base compiler. As one is needed to bootstrap Pkgsrc (Ravenports is self-contained in this case), ‘yum groupinstall “Development Tools”‘ was used to make the system capable of building software. On OmniOSce there is also no base compiler; ‘pkg install gcc10’ took care of making the system fit to continue on.

With all systems the first step after preparing the environment – i.e. making a compiler available if it’s missing as well as configuring SSH – was to fetch pkgsrc-2021Q3.tar.xz and decompress it to /usr. Then the typical bootstrap was run (except on NetBSD where this is not required) and the PATH environment variable modified to include /usr/pkg/sbin and /usr/pkg/bin.

The next step was always to build pkgtools/mksandbox as well as pkgtools/pbulk and to attempt a bulk build. Since this failed for one reason or another in almost all cases, plan b was to at least run (b)make in /usr/pkgsrc, letting the system build as many packages as possible. A list of packages was created and the build started over after declaring all the licenses acceptable (otherwise quite a number of packages that can be build are skipped).

With Ravenports it’s doing the bootstrap as explained in the documentation for each supported OS and eventually running ‘ravenadm build-everything’. The results for Solaris / illumos were taken from a list of the packages in the official repository at that time. The reason is that I did not have enough spare time to try and install the specific old version of Solaris (from 2009!) required in the VM.

Results

Here are the results of what is likely one of the more comprehensive evaluations of Pkgsrc ever done. Package building on all the platforms took almost two months’ time. The various criteria have been grouped together into the “OS support”, “General” and “Technical evaluation” groups. Unfortunately the tables did not fit well either in Gemtext or in blog-style HTML and are therefore available as images.

Operating System support

Pkgsrc vs. Ravenports: Operating system support

As mentioned above, keep in mind that the amount of packages that built with Pkgsrc is the minimum count of buildable ones. Where the numbers look particularly low, that’s usually due to a very basic dependency package being broken, cutting off a substantial part of the tree (e.g. Python on DragonFly BSD and pkgconf on illumos).

Disregarding Linux and illumos support (as those are out of scope for our use case), both score almost evenly. Pkgsrc comes out very slightly ahead [1:0] of Ravenports due to it’s excellent integration into one of our main platforms (NetBSD).

If both Linux and illumos are to be taken into account, too, Pkgsrc’s score remains the same as two additional supported platforms with two failed bulk builds neutralize each other. Ravenports however gains two points for Linux support including bulk builds and one more point for illumos support (since it can bulk-build for but not on illumos, I’d give a 0 instead of a + rating in this case). This would place Ravenports ahead [1:3].

General criteria

Pkgsrc vs. Ravenports: General

While the former section ended up very surprising for me (having taken for granted that full bulk build support was the rule rather than the exception!), this one is very much what you’d expect.

Pkgsrc, being an established project with a long history and a lot of contributors, can show its strengths to the fullest here. The newcomer does not perform too badly actually, but despite getting ahead in one test (package freshness) it doesn’t stand the slightest chance against the veteran. Pkgsrc wins with a huge lead [12:-1] and Ravenports leaves the field beaten and bruised in crushing defeat.

Technical evaluation

Pkgsrc vs. Ravenports: Technical evaluation

While it looked like the match was already over after the previous round, this one is where being slow and old-fashioned despite being mature puts you at risk. So here’s where the youngster can shine with its modern-day approaches and new tricks! I expected Ravenports to clearly beat Pkgsrc this time. It did – but it even managed to make this round look like roles being inverted compared to the previous one…

This time it’s Pkgsrc that can win only one test (popularity of languages used). Ravenports however manages to bring out the big guns and eventually leaves Pkgsrc utterly destroyed by an even higher margin [-3:11] than it lost the previous round.

Final results

Pkgsrc vs. Ravenports: Final results

For me this duel between Pkgsrc and Ravenports has been more interesting than I would have thought. After Pkgsrc messed up the jump start that I was expecting, things began to look like a more evenly matched comparison. I still would have lost my money as I’d have bet Pkgsrc to win at least by a tiny margin.

It’s a bit weird but at the same time probably fitting to see that both contenders have such a diverse set of strengths and weaknesses – and that their comparison eventually even ended [10:10] in a draw!

But that’s kind of delivering old news, since I’m writing about the comparison as of late September. In October, Ravenports gained official support for NetBSD, which would shift the results in Ravenports favor [10:14]. It could provide more insights to give Pkgsrc a try again when the Q4 release happens. I haven’t decided, yet. Maybe I’ll do that.

Conclusion

As stated above, Pkgsrc is NetBSD’s ports system. Our tests show that you need to stress this: It’s not simply a statement about its origin. It’s a statement about its main purpose.

NetBSD is where Pkgsrc development takes place. Other operating systems are invited to participate, but maintaining excellent support for any other OS is a huge task. Before I took a closer look at this, I didn’t know that even Joyent (the company behind SmartOS which uses Pkgsrc as its official means of package management) do maintain a downstream fork of Pkgsrc!

They have employees who are doing paid work on Pkgsrc – and obviously they still do not manage to keep upstream Pkgsrc in good enough shape for themselves (leading to the less than impressive number of packages that currently work on illumos).

Ravenports is more like the new kid on the block. It has proven to be a serious effort (being in active development for over 5 years now) but has yet to find its niche. Despite it being technically superior in many ways, it’s held back by the very low number of contributors.

Both solutions can be used for cross-platform package management. So which one should you choose? While they can be used for the same scenario, their goals are very different. Pkgsrc is meant to run even on the most exotic, limited and obsolete platforms and is willing to pay a price for that. Ravenports in contrast focuses entirely on modern hardware and can take advantage of their superior capabilities. Therefore there’s no “one size fits all” in this case, it really depends on what you want to do.

Need to build ports directly? Pkgsrc’s the answer here. Have to install software on AIX? Use Pkgsrc. Running a SPARC64 machine? Pkgsrc again. If however you need the latest software versions, go for Ravenports. Are reliable package builds across multiple platforms what you’re looking for? Give Ravenports a try. Don’t want to invest into build clusters for acceptable performance? Go with Ravenports if you can.

Consider your use case, compile a table of requirements. Then see which solution fits you best. But whichever you pick, always consider that you might have to do some work of your own (and please contribute it back!).

Cross-platform package building: Pkgsrc vs. Ravenports (1/2)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/cross-platform_package_building_pt1.gmi

This is the first of a two articles on cross-platform package management / package building. It covers the basics by discussing why it is actually surprisingly (to many people) difficult to do and what some of the problems are. It also takes a quick look at some strategies to solve the problem.

Package management: Closely platform-dependent

One of the strengths of today’s Unix-like operating systems is that they offer proper package management. This makes installing and maintaining software both simple and convenient. If we are talking homogeneous environments, this is! As soon as you have a heterogeneous environment to work in, things get complicated rather quickly. Packages and package systems are usually closely tied to their platform. It’s not a coincidence that people talk about e.g. “RHEL-like” distributions and “RPM-based” ones kind of interchangeably. The package manager is so close to the core of an OS or a distribution that it can be used to refer to that synonymously.

It’s usually not the package manager itself (unless it is a special one offering features that define a whole distribution that uses it like Gentoo’s portage that allows for fine-grained custom building of applications thanks to its USE flags among other things or NixOS’s nix which as a purely functional package manager that supports concurrent installations of multiple versions, atomic upgrades, etc.). Whenever an application (or something else like a theme pack, documentation, etc.) is to be packaged, the maintainer has to make a couple of decisions which maintainers of the same software for another OS / distribution might have another opinion on.

Different package naming schemes

One of the first things to mind is that package names may differ between operating systems / distributions. A popular example is the Apache webserver being available as httpd in Linux distributions based on Red Hat’s while in Debian-based ones it’s known as apache2. FreeBSD uses the name apache24 and OpenBSD calls the same webserver apache-httpd. But that’s only the names, any configuration management system (or even every admin by hand!) can handle that easily. The software is the same after all, right? Yes and no!

Different configuration options and structure

While it’s all built from the code that is released by the same upstream project, all the platforms organize the software differently. Sticking to the Apache example, it’s pretty well known that Debian-based distributions use a mechanism called “sites-enabled” while others like Red Hat do not. This means to either embrace multiple schemes that are native to the platforms which you use or having to create your own and bent the default installations on all platforms to use that. It’s not such an uncommon thing to harmonize configuration and doing it is not incredibly hard.

It comes at a price, though. Hired a new admin? You could probably expect him or her to be familiar with the standard scheme. But if you’re using a custom one, the new employee will need time to become familiar with it. You also don’t do it once and be done. The default configuration is likely to change over time. In case of our webserver for example the recommended ciphers for TLS encryption may change. If you use the default configuration you’ll probably get important changes like this for free when updating. Forsaking it and doing your own means more homework for you to keep the configuration in good shape.

Different paths

Speaking of which: On FreeBSD you will even find the configuration files in another place (/usr/local/etc/apache24) while on Linux it’s commonly something below /etc (like /etc/apache2 or /etc/httpd). Other things like databases are frequently in different place, too. On FreeBSD it’s /var/db/mysql for example while on Linux it’s usually /var/libexec/mysql. By itself that’s just another small detail to take into account. But those add up and should not be neglected entirely.

Different compile-time options

And even worse: The package maintainers for each platform make decisions on compile-time options! So the resulting software will differ even if you go the extra mile and configure them alike with your custom runtime configuration scheme! Even seemingly basic things should not be just taken for granted. Once upon a time, the Apache webserver often came _without_ SSL support – sometimes there was an extra package which had it enabled for people who needed that functionality! Sometimes you had to build the software yourself (or use ports that let you set or unset the option as you please). On Debian, the Apache package has Lua support enabled but not that for LuaJIT. On FreeBSD both are disabled by default. The FreeBSD port offers more than 130 (!) options – it’s not hard to see how much of a difference the choice of the package maintainer can make for more advanced software!

Patches

While probably any reader will understand pretty well by now that our topic is rather far away from all peaceful unicorns and sunny weather, it gets worse still. It’s not uncommon that package maintainers choose to apply patches instead of using the code as upstream provided it. This may be due to an incompatibility (maybe some dependency that this OS / distro ships is too old or too new for this software and so a patch is required to make it play along nicely). It may be because the maintainer feels that a fix that was not deemed important enough to warrant another release should still be applied. It could be because additional features not supported upstream are desired (many maintainers chose to ship a pretty popular but unofficial additional MPM called ITK for example). Or it could because of any number of other good or bad reasons. Therefore software might even differ between various platforms if the exact same compile-time options were chosen…

Versions

And because we of course saved the best for last, the biggest problem is an even more demanding one: Package versions… Not only do the various operating systems / distributions update to newer versions at their own pace, but they may or may not backport fixes or newer features into the packages that they release! Keeping track of this is already a major hassle for a couple of programs – and it becomes a downright daunting task if you need to do it for many! And you have to. It’s _not_ optional. Why? Because newer versions of your software might introduce newer features or configuration directives that you definitely want to use. However you cannot simply enable them for all of your servers as the older versions will probably refuse to even start due to invalid (unknown) settings!

Newer versions may also deprecate or remove previously supported features. There are features that may only be available on certain platforms (maybe additional dependencies are required which are not ported to every platform that you use). All this kind of fun stuff that can totally ruin your weekend when it eventually bites you despite you having been lucky for a long time before it.

Cross-platform package building strategies

The more diverse your environment is, the more the consequences of what was just scratched above are going to make your job look like one of the inner circles of hell. What’s the best way out of this misery?

Well, what about compiling the most important software yourself and deploying it to e.g. /opt? This is technically very much possible, but is it feasible in large scale? You’re almost guaranteed to drown eventually. Don’t give in to the temptations of going down this road! This way lies insanity.

If you’ve only got a few different systems in use and not too many complex programs, you might get away with careful planning and careful configuration management. It won’t be pretty but it’s possible to do. Got several different systems that you need to support? Do yourself a favor and find another solution.

There is in fact a proper solution to this: Using a package framework that supports multiple platforms. Doing so comes with its own set of challenges and pains, but they are much easier to bear. You probably have to learn to use a new package management tool. (Depending on your choice) you might need to understand the concept of a ports tree if you are not already familiar with it. But doing so you will be able to use packages of the same version, built with the same compile-time options (or at least very close if various platforms force diverging settings) and so on across your entire landscape!

Sounds too good to be true? Let’s put two options which claim to be able to do just that to the test! For the Advance!BSD project we plan to use at least four different operating systems (for more information see: Advance!BSD – thoughts on a not-for-profit project to support *BSD pt. 1 and pt. 2). Using the native packages on each one is basically out of question. Especially since we anticipate that we’ll need to add some software packages of our own to the mix and totally lack the manpower to maintain that across four package systems.

What’s next?

The next article will introduce Pkgsrc and Ravenports and present the results of a 2 months evaluation of Pkgsrc on four BSD operating systems + Linux and illumos. It will also compare the advantages and disadvantages of both contenders in heterogenous environments.

Updating FreeBSD 4.11 (4/4) – Reflecting radical resurrection

In the first post of this mini series I wrote about legacy systems and installing FreeBSD 4.11. The second one shows how to configure the fresh system for remote access, bootstrap Pkgsrc, install Subversion to checkout FreeBSD code and update the system to the stable branch. And part three mainly deals with upgrading OpenSSH and the compilers. This post details some more updates until we reach the final state that’s possible with such an old system (without resorting to extreme means).

Planting a new tree

So far we’ve built some packages from 2013 and before. Using a current pkgsrc tree won’t work – the various pkgsrc tools that our system has are too old. It might not be too big a step but we can use a tree from the second half of 2014. Of course the newer SSH that we built before is not currently in the path so we need to create a temporary symlink before we can use CVS again:

# ln -s /usr/local/temp/bin/ssh /usr/local/pkgsrc/bin/ssh
# rehash

# cd /usr/pkgsrc
# cvs -danoncvs@anoncvs.netbsd.org:/cvsroot get -rpkgsrc-2014Q3 -P pkgsrc
# mv pkgsrc 14
# rm /usr/local/pkgsrc/bin/ssh

What the system looks like package-wise at the beginning of part 4

Most of pkgsrc’s tools make use of NetBSD’s compatibility library. Unfortunately the version that comes with the new pkgsrc tree won’t build anymore on an OS as old as FreeBSD 4.11. Same thing for libfetch. But the newer tools will work with older versions of that libs, too. So let’s prepare those two – libfetch need’s some more love to build:

# cd /usr/pkgsrc/13/pkgtools/libnbcompat
# bmake

# cd /usr/pkgsrc/13/net/libfetch
# cp Makefile Makefile.bak
# sed '14i\\
CFLAGS=         -Wno-error' Makefile.bak > Makefile
# bmake

As a next step we’re going to do two updates. Yes, in theory we could use “bmake update” to update packages. We will not do that. The reason is that we needed to abuse pkgsrc quite a bit so far by mixing package versions from various trees. Since “bmake update” is a destructive command (it will happily uninstall programs as well as packages depending on them!) this can lead to all sort of fun things like unresolvable dependencies and such.

If you like pain, go ahead. I’ve been there and I can confirm that it does work for some packages. For a lot of them actually. But in those cases where it doesn’t, it tends to do so much damage that you’re better off starting over than trying to fix things… That’s why I’ll show you a safer method instead: Build a package and update via pkg_add! Also it really starts to show how old the system is that we’re trying to build rather new packages on. More and more of them require some trickery to persuade them to build – but hey, we’re doing a gross thing here, anyway. So there’s no real reason to complain!

# cd /usr/pkgsrc/14/pkgtools/pkg_install
# bmake extract
# rm -r work/libnbcompat/*
# rm -r work/libfetch/*
# cp -R /usr/pkgsrc/13/pkgtools/libnbcompat/work/libnbcompat-20120702/* /usr/pkgsrc/14/pkgtools/pkg_install/work/libnbcompat/
# cp -R /usr/pkgsrc/13/net/libfetch/work/libfetch-2.34/* /usr/pkgsrc/14/pkgtools/pkg_install/work/libfetch/
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/14/packages/All/pkg_install-20130902nb1.tgz

# cd /usr/pkgsrc/14/pkgtools/bootstrap-mk-files
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/14/packages/All/bootstrap-mk-files-20140516.tgz

We made it so far, now let’s make a daring move and just download the latest stable pkgsrc tree – released in January 2017:

# cd /usr/pkgsrc
# fetch http://cdn.netbsd.org/pub/pkgsrc/stable/pkgsrc-2016Q4.tar.bz2
# tar xvjf pkgsrc-2016Q4.tar.bz2
# rm pkgsrc-2016Q4.tar.bz2
# mv pkgsrc 16

Updating pkgsrc tools

Since mid 2014, pkgsrc makes use of a new package, cwrappers. During my test run I somehow managed to just get this package built. Despite taking notes I have no idea what I did to just make it work! It must have been something that looked like a dead end (which is why I didn’t include it in my notes) but somehow provided “getline”… I tried to get it working again for almost one whole Sunday but for the life of me couldn’t find out what I previously did… In the end I gave up and tried to find another solution. I found one but while it is way more complex it at least means that I got rid of that nasty blocker again:

# cd /usr/pkgsrc/16/pkgtools/cwrappers
# bmake extract
# rm -r work/libnbcompat/*
# cp -R /usr/pkgsrc/13/pkgtools/libnbcompat/work/libnbcompat-20120702/* /usr/pkgsrc/16/pkgtools/cwrappers/work/libnbcompat/
# cp work/cwrappers-20161125/mi_vector_hash.c work/cwrappers-20161125/mi_vector_hash.c.bak
# cp work/cwrappers-20161125/fixup-libtool.c work/cwrappers-20161125/fixup-libtool.c.bak
# sed 's/stdint.h/inttypes.h/' work/cwrappers-20161125/mi_vector_hash.c.bak > work/cwrappers-20161125/mi_vector_hash.c
# sed 's/stdint.h/inttypes.h/' work/cwrappers-20161125/fixup-libtool.c.bak > work/cwrappers-20161125/fixup-libtool.c
# cp /usr/pkgsrc/14/pkgtools/cwrappers/files/bin/getline.c /usr/pkgsrc/16/pkgtools/cwrappers/work/cwrappers-20161125/getline.c.bak
# sed 's/ssize_t/size_t/' work/cwrappers-20161125/getline.c.bak > work/cwrappers-20161125/getline.c
# cp work/cwrappers-20161125/common.h work/cwrappers-20161125/common.h.bak
# sed '107i\\
size_t  getline(char **, size_t *, FILE *);' work/cwrappers-20161125/common.h.bak > work/cwrappers-20161125/common.h
# cp work/cwrappers-20161125/Makefile work/cwrappers-20161125/Makefile.bak
# sed '14i\\
LIB_SRCS+=      getline.c' work/cwrappers-20161125/Makefile.bak > work/cwrappers-20161125/Makefile
# bmake install clean clean-depends

Phew! Fortunately the next few updates are straight forward:

# cd /usr/pkgsrc/16/pkgtools/bootstrap-mk-files
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/bootstrap-mk-files-20160908.tgz

# cd /usr/pkgsrc/16/devel/bmake
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/bmake-20150505.tgz

# cd /usr/pkgsrc/16/net/tnftp
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/tnftp-20151004nb1.tgz

Next is another one that requires some patching:

# cd /usr/pkgsrc/16/pkgtools/digest/
# bmake extract
# cp work/digest-20160304/sha3.h work/digest-20160304/sha3.h.bak
# cp work/digest-20160304/keccak.c work/digest-20160304/keccak.c.bak
# cp work/digest-20160304/keccak.h work/digest-20160304/keccak.h.bak
# cp work/digest-20160304/sha3.c work/digest-20160304/sha3.c.bak
# sed 's/stdint.h/inttypes.h/' work/digest-20160304/sha3.h.bak > work/digest-20160304/sha3.h
# sed 's/stdint.h/inttypes.h/' work/digest-20160304/keccak.c.bak > work/digest-20160304/keccak.c
# sed 's/stdint.h/inttypes.h/' work/digest-20160304/keccak.h.bak > work/digest-20160304/keccak.h
# sed 's/stdint.h/inttypes.h/' work/digest-20160304/sha3.c.bak > work/digest-20160304/sha3.c
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/digest-20160304.tgz

Updating installed packages

Let’s update gettext first as a lot of packages need that one; xz is one of the packages that is linked against the old one and since libintl received a soname bump, it needs to be rebuilt. Since we want to update it anyway that’s not too bad. But there are other packages that we cannot update which depend on the old lib. So we’ll have to create a symlink to satisfy their need, too:

# cd /usr/pkgsrc/16/devel/gettext-lib
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gettext-lib-0.19.8.1.tgz

# cd /usr/pkgsrc/16/archivers/xz
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/xz-5.2.2.tgz

# ln -s /usr/local/pkgsrc/lib/libintl.so.9 /usr/local/pkgsrc/lib/libintl.so.7
# cd /usr/pkgsrc/16/devel/gettext-tools
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gettext-tools-0.19.8.1.tgz

Next in line is some more typical build dependencies:

# cd /usr/pkgsrc/16/devel/libtool-base
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/libtool-base-2.4.2nb13.tgz

# cd /usr/pkgsrc/16/devel/m4
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/m4-1.4.17.tgz

# cd /usr/pkgsrc/16/devel/bison
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/bison-3.0.4nb3.tgz

Just a few more packages and we’ll have updated most packages that can be updated (a few like zip and nbpatch can’t):

# cd /usr/pkgsrc/16/shells/bash
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/bash-4.4.005.tgz

# cd /usr/pkgsrc/16/lang/perl5
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/perl-5.24.0.tgz

# cd /usr/pkgsrc/16/devel/autoconf
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/autoconf-2.69nb7.tgz

# cd /usr/pkgsrc/16/devel/gmake
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gmake-4.1nb3.tgz

Rebuilding the compiler

First we need to update the two math libraries (and create another symlink so we can go on compiling):

# cd /usr/pkgsrc/16/devel/gmp
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gmp-6.1.2.tgz
# ln -s /usr/local/pkgsrc/lib/libgmp.so.13 /usr/local/pkgsrc/lib/libgmp.so.11

# cd /usr/pkgsrc/16/math/mpfr
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/mpfr-3.1.5.tgz

This unfortunately breaks the compiler. But we can still resort to the old GCC3 to build GCC4 again, right? Right:

# cp /usr/pkgsrc/13/distfiles/gcc-4.4.7.tar.bz2 /usr/pkgsrc/16/distfiles
# cp /root/.cshrc /root/.cshrc.bak
# sed 's:pkgsrc/gcc44:temp/gcc34:' /root/.cshrc.bak > /root/.cshrc
# source /root/.cshrc
# cc -v

While we probably still don’t need Object-C or Java we could in fact build GCC with them this time. Java requires Python2.7 installed but that can actually be built from the 2014 tree! The problem is that building Java requires more RAM than is available on 32 bit machines and will for that reason fail. However Java is deactivated by default for GCC 4.4 in the 2016 tree. So let’s just get rid of our custom options, build the default package and set the correct path again:

# cp /usr/local/pkgsrc/etc/mk.conf /usr/local/pkgsrc/etc/mk.conf.bak
# sed '/PKG_OPTIONS.gcc44/d' /usr/local/pkgsrc/etc/mk.conf.bak > /usr/local/pkgsrc/etc/mk.conf
# cd /usr/pkgsrc/16/lang/gcc44
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gcc44-4.4.7nb7.tgz

Almost everything updated!

Now we only have to restore the correct path and then we have the GCC4 back (with a newer patch level):

# cp /root/.cshrc.bak /root/.cshrc
# source /root/.cshrc
# cc -v
gcc version 4.4.7 (GCC)

Modern OpenSSH

There’s one more package to build that needs a bit of care: Pkgconf. It’s a simpler replacement for the older pkg-config but it won’t work out of the box for us:

# cd /usr/pkgsrc/16/devel/pkgconf
# bmake extract
# cp work/pkgconf-1.0.1/libpkgconf/stdinc.h work/pkgconf-1.0.1/libpkgconf/stdinc.h.bak
# cp work/pkgconf-1.0.1/getopt_long.h work/pkgconf-1.0.1/getopt_long.h.bak
# sed 's/stdint.h/inttypes.h/' work/pkgconf-1.0.1/libpkgconf/stdinc.h.bak > work/pkgconf-1.0.1/libpkgconf/stdinc.h
# sed 's/stdint.h/inttypes.h/' work/pkgconf-1.0.1/getopt_long.h.bak > work/pkgconf-1.0.1/getopt_long.h
# bmake install clean clean-depends

Finally the time has come to do what I wanted to do in the first place, provide a recent version of OpenSSH! Of course it’s also necessary to generate new host keys once more. And then, just to prove everything works when the machine boots, let’s just restart the machine after adjusting the sshd path:

# cd /usr/pkgsrc/16/security/openssh
# bmake install clean clean-depends
# rehash
# ssh -V

# ssh-keygen -f /usr/local/pkgsrc/etc/ssh/ssh_host_rsa_key -N '' -t rsa
# ssh-keygen -f /usr/local/pkgsrc/etc/ssh/ssh_host_dsa_key -N '' -t dsa
# mkdir -p /usr/local/pkgsrc/run

# cp /etc/rc.conf /etc/rc.conf.bak
# sed 's:temp/sbin:pkgsrc/sbin:' /etc/rc.conf.bak > /etc/rc.conf
# shutdown -r now

Generating new host keys for OpenSSH

Now we need to remove the vierelf entry in ~/.ssh/known_hosts before we connect again. Doing so in verbose mode even shows that the 4.11 box now has a newer version of OpenSSH installed that the FreeBSD 11 workstation that I use to connect to it! 😀

FreeBSD 4.11 running a newer OpenSSH than my FreeBSD 11.0 workstation!

Conclusion

FreeBSD 4.11 is really, really, really old now. But you can get surprisingly far in running somewhat modern software on it – more recent software at least than I initially thought would be possible! And you? What was your bet? Would you have guessed that I’d make it up to the 2016Q4 pkgsrc tree and even install the latest version of OpenSSL and OpenSSH?

Here’s a little summary of some important program updates:

binutils 2.12.1 (2002) -> binutils 2.17 (2006)
perl 5.005 (1998) -> perl 5.8 (2002) -> perl 5.18 (2013) -> perl 5.24 (2016)
GCC 2.95.4 (2001) -> GCC 3.4.6 (2006) -> GCC 4.4.7 (2012)
OpenSSH 3.5 (2002) -> OpenSSH 4.6 (2007) -> OpenSSH 7.3 (2016)

Not too bad, eh? The notable exception here is binutils. Newer versions would probably be possible but there’s a gap in pkgsrc – which stuck with 2.17 for a long time and then directly moved to 2.22 which no longer builds on FreeBSD 4.11. GCC 4.5.3 does build BTW but something goes sideways and the comparison of stage 2 and 3 fails for quite some files.

I’ve met my initial goal to provide a newer version of OpenSSH, surpassing all expectations that I had. There’s room for more of course but that’s not worth another post. I’m going to add sudo and since Python 2.7 can be built it might even be possible to manage the 4.11 servers using salt-ssh (the ordinary SaltStack doesn’t work as it requires ZeroMQ which looks like it cannot be built)! We have a recent version of bash and can thus do some pretty nifty things with the right .bashrc.

This whole adventure took far longer than I had anticipated – a bit over a month instead of the intended two weekends! But that was mostly because I decided to start over with a clean system several times to ensure that everything works as I wrote it down here (and because GCC4 simply takes so long to build on the only spare machine that I had for this…). But it has been an interesting ride and I don’t regret spending some time on the legendary FreeBSD 4.11!

Oh, and my special thanks to everybody involved with Pkgsrc! I usually don’t have much use for NetBSD but Pkgsrc is extremely useful. I might use it in the future on other systems (like Linux), too. And thanks to you for reading. I hope that you enjoyed it as well!

Updating FreeBSD 4.11 (3/4) – Neophyte’s notorious necromancy

The first post of this mini series was about legacy systems in general and about what installing the old FreeBSD 4.11 is like. In the second one I showed the initial configuration of the system, how to SSH into it despite the obsolete DSA host key and how to bootstrap pkgsrc, NetBSD’s portable ports tree. I also covered the installation of SVN, checking out of the 4.11-STABELE code and updating the system. This post will cover installing newer software.

Any bets?

So far we have a pkgsrc tree from mid 2007 and things seem to be working. However that’s pretty close to 4.11’s release in 2005 and thus not too amazing. Working with such an old system there are plenty of cases which mean “game over”. Here are just three errors of that kind which you can encounter trying to build more modern software:

/usr/libexec/elf/ld: cannot find -lpthread

There’s no modern pthreads available on 4.11. Game over.

/usr/include/sys/resource.h:58: error: field 'ru_utime' has incomplete type

We’ll have to do with very old system headers missing a lot of what we take for granted today. Game over again.

fileio.o(.text+0x354): undefined reference to `towupper'
collect2: ld returned 1 exit status

Sorry, that ancient libc that we have on our system doesn’t provide that symbol. Game over yet again.

How far do you think can we take it in building and installing more recent software? Make a guess now and see if you were right! To be honest I was not expecting the end result. Not at all. So let’s get back to work!

In for a screening

We’re going to compile a lot of stuff this time – building SVN and dependencies was just a warm up. And what do you do when you’re building stuff remotely over SSH? You’re doing so in a screen or tmux session of course. Neither is part of the base system so we’ve got to build one. Tmux was not yet available in 2007 so it’s not too hard a choice:

# cd /usr/pkgsrc/07/misc/screen
# bmake install clean clean-depends
# rehash
# screen

GNU screen started up and ready

If you don’t know screen do some reading because you will want to start using it (or rather the superior tmux). It basically allows you to detach from a session and reconnect later – and your programs will continue running on the remote system even while you’re logged out. You can also resume the session from another terminal or computer, share sessions, etc. And that’s just one of the things that it does. There are other features like allowing you to have multiple shell instances in just one terminal between which you can switch back and forth (think tabs of a browser) and a lot more. Should you not like this (what’s wrong with you?!), fine. Don’t install screen. It’s optional.

Replacing the front door lock

Now it’s time to take care of the main problem of our system: That darned version 3.5 of OpenSSH! Let’s build whatever our pkgsrc tree has to offer:

# cd /usr/pkgsrc/07/security/openssh
# bmake install clean clean-depends
# rehash
# ssh -V
OpenSSH_4.6p1, OpenSSL 0.9.7d-p1 17 Mar 2004

Still far from a modern version of OpenSSH but also a lot better. And the best thing: It supports RSA keys. Let’s generate host keys with this newer SSH and make it the version that FreeBSD launches during startup:

# ssh-keygen -f /usr/local/temp/etc/ssh/ssh_host_key -N '' -t rsa1
# ssh-keygen -f /usr/local/temp/etc/ssh/ssh_host_rsa_key -N '' -t rsa
# ssh-keygen -f /usr/local/temp/etc/ssh/ssh_host_dsa_key -N '' -t dsa
# mkdir -p /usr/local/temp/run
# echo 'sshd_program="/usr/local/temp/sbin/sshd"' >> /etc/rc.conf

Ok, everything is in place. We could reboot now – or just kill off the old daemon and launch the new one. Let’s first look for SSHD and see which PID it has (this of course varies from system to system!):

# ps aux | grep sshd

Replacing SSHD

Got it? Great, let’s kill it (your SSH connection is maintained by a child and it’s generally save to kill the parent. You won’t lose your SSH connection!), start the new one and ensure that it’s running:

# kill [PID on your system]
# /usr/local/temp/sbin/sshd
# ps aux | grep sshd

What’s this? It looks like it’s not running! Yes, it looks like it but actually it should be running… Let’s grep again:

# ps aux | grep local

This does return one process – and trust me it’s actually our new sshd. What’s happening here is this: The output of ps is truncated because more wouldn’t fit on the screen. And only that data is handed to grep! So the process with the name /usr/local/temp that we found (see the screenshot above) is actually /usr/local/temp/sbin/sshd with the last part of it cut off… This is why grep doesn’t find “sshd”. There’s a funny way to fix this, though: Maximize your terminal emulator so that more space is available. Then grep will find sshd!

Now we can quite the old SSH session so we can make one with the new server. We can even keep our screen session open, but we need to detach from it by pressing CTRL-A and then D before we logout from vierelf:

[detached]

# logout
> exit
Connection to 192.168.1.5 closed.

Time to edit your known hosts and get rid of the former host key for vierelf or else you’ll see that scary SSH warning when you try to login again. Oh, and you can leave out that compatibility option from now on – which is a major step ahead! When you’re back in, you can resume the screen session:

% ssh kraileth@192.168.1.5
> su -
# screen -r

Connecting to the new SSH server (debug mode)

Compiler: from antediluvian to ancient

Alright. Currently we have the last version of the second generation of GCC on our system. We totally need to get our hands on something newer. How about updating the last version of generation three? Let’s try that! We only want the C and C++ compilers. Fortran is deactivated by default for this version (it would need GMP installed and the version of GMP that’s in the tree requires GCC3. It’s a good idea to avoid that potential circular dependency). However Java and Object-C are activated. There’s no need to waste time on them, they should be deactivated as well. The following sed command may look a bit complex, but it’s not that bad. Just copy all three lines that make up that single command and you’re good to go:

# cd /usr/pkgsrc/07/lang/gcc34
# cp Makefile Makefile.bak
# sed -e '64,65d' -e '63a\\
BUILD_JAVA?=    NO' -e '63a\\
BUILD_OBJC?=    NO' Makefile.bak > Makefile
# bmake install clean clean-depends

After installing that newer GCC, the path needs to be changed again so that the system picks it up instead of the older system compiler:

# vi /root/.cshrc

Prepend the following path to the PATH variable:

/usr/local/temp/gcc34/bin

Now let’s log out and in again and see if the new compiler is available:

# exit
[screen is terminating]
# logout
> su -
# screen
# cc -v
[...]
gcc version 3.4.6

Updating pkgsrc

Since we also have a more recent OpenSSH now, we can checkout a newer copy of pkgsrc from CVS! That takes a while, be patient. Even after it is finished downloading (and you see no new lines on the screen) it will still take some time to clean things up. This is normal and you have to wait a little longer. Don’t CTRL+C it as that would leave your tree in bad shape!

# cd /usr/pkgsrc
# cvs -danoncvs@anoncvs.netbsd.org:/cvsroot get -rpkgsrc-2009Q4 -P pkgsrc
# mv pkgsrc 09

Thanks to the newer SSH: CVS works now, too!

We’ll need some ports from there later. But since we have GCC 3 available now we can also grab an even newer copy and primarily use that one:

# cvs -danoncvs@anoncvs.netbsd.org:/cvsroot get -rpkgsrc-2013Q2 -P pkgsrc
# mv pkgsrc 13

We’re going to start a fresh environment, using only GCC (and sshd) from the old one. To do so we first bootstrap the pkgsrc from 2013 into a new directory:

# mkdir /usr/local/pkgsrc
# cd /usr/pkgsrc/13/bootstrap
# ./bootstrap --prefix=/usr/local/pkgsrc --varbase=/usr/local/pkgsrc

The next step is to adjust the path variable so that the binaries from the new location are being used. To do so we need to replace /usr/local/temp with /usr/local/pkgsrc for both sbin and bin. Don’t change the compiler path, though! GCC 3 will remain in temp. After logging out and back in, screen is no longer in PATH so we need to execute it with the absolute path:

# cp /root/.cshrc /root/.cshrc.bak
# sed -e 's:temp/bin:pkgsrc/bin:' -e 's:temp/sbin:pkgsrc/sbin:' /root/.cshrc.bak > /root/.cshrc
# exit
# logout
> su -
# /usr/local/temp/bin/screen

Cherry-picking dependencies

This gives us a way to easily build software from 2013. Let’s continue on by fetching some source tarballs by hand that are no longer available on the mirrors that pkgsrc knew for them:

# cd /usr/pkgsrc/09/distfiles
# fetch http://ftp.cc.uoc.gr/mirrors/NetBSD/packages/distfiles/binutils-2.17.tar.gz
# fetch http://ftp.cc.uoc.gr/mirrors/NetBSD/packages/distfiles/pkg-config-0.23.tar.gz

The following part is not too interesting: We’re going to build the dependencies in preparation for the next big step. In general we try to build the newest version possible (2013) but resort to old (2009) or even older (2007) where necessary if newer versions don’t build for various reasons:

# cd /usr/pkgsrc/13/converters/libiconv
# bmake install clean clean-depends

Zip from 2009 and onwards is incompatible with FreeBSD 4.11’s libc. And the 2007 version expects tar in a location where there’s none on our system. Instead of building tar we can safely symlink it:

# ln -s /usr/bin/tar /usr/local/pkgsrc/bin/tar
# cd /usr/pkgsrc/07/archivers/zip
# bmake install clean clean-depends

The binutils are a special case. The port normally builds the programs of which it consists with a prefix so they don’t get in the way of the system binaries. Since we actually want to use them instead of the old stuff from the base system, we need to get rid of that prefix:

# cd /usr/pkgsrc/09/devel/binutils
# bmake GNU_PROGRAM_PREFIX='' install clean clean-depends
# rehash
# ld -v
GNU ld version 2.17

The next few are trivial:

# cd /usr/pkgsrc/09/devel/gettext-tools
# bmake install clean clean-depends

# cd /usr/pkgsrc/13/devel/m4
# bmake install clean clean-depends

# cd /usr/pkgsrc/09/devel/bison
# bmake install clean clean-depends

The bash port from 2013 would draw in a newer version of gettext which would not build. But bash can actually be built with the old one, too. So we have to make a simple change in the buildlink file for gettext in 2013’s pkgsrc tree:

# cd /usr/pkgsrc/13/devel/gettext-lib
# cp buildlink3.mk buildlink3.mk.bak
# sed 's/0.18/0.14/g' buildlink3.mk.bak > buildlink3.mk

With that change the next port can be built:

# cd /usr/pkgsrc/13/shells/bash
# bmake install clean clean-depends

Next in line is perl. The 2013 port would however build with dtrace support by default – which was of course not available on 4.11. Therefore it needs to be switched off by making an addition to the pkgsrc config file:

# vi /usr/local/pkgsrc/etc/mk.conf

Add the following line at the end of the file (but above .endif):

PKG_OPTIONS.perl=       -dtrace

Now let’s build the last few dependencies:

# cd /usr/pkgsrc/13/lang/perl5
# bmake install clean clean-depends

# cd /usr/pkgsrc/13/archivers/xz
# bmake install clean clean-depends

# cd /usr/pkgsrc/09/devel/autoconf
# bmake install clean clean-depends

Compiler: from ancient to old

With this all dependencies from earlier than 2013 in place we are good to go for the biggest update. We’re still not interested in Java and Object-C, so let’s edit pkgsrc’s configuration again:

# vi /usr/local/pkgsrc/etc/mk.conf

and add one more line (e.g. after the perl one):

PKG_OPTIONS.gcc44=      -gcc-java -gcc-objc

Building the newer version of GCC means building two more dependencies as well, one of which is libgmp. GMP is the first package so far that uses C++ and in fact our C++ compiler has been broken the whole time. Luckily a symlink can heal it and another one will make GCC happy so that we can finally build it – which takes quite a bit of time (I’ve seen the compilation stop at one point and I’m not sure what happens there. But just calling bmake again will eventually complete the build process!):

# ln -s /usr/local/pkgsrc/lib/libiconv.so.7 /usr/lib/libiconv.so.7
# ln -s /usr/local/temp/gcc34/lib/libgcc_s.so.1 /usr/lib/libgcc_s.so.1
# cd /usr/pkgsrc/13/lang/gcc44/
# bmake install clean clean-depends

Once it’s build, we need to change our PATH so that the newer GCC is the primary compiler:

# mv /root/.cshrc /root/.cshrc.bak
# sed 's:temp/gcc34:pkgsrc/gcc44:' /root/.cshrc.bak > /root/.cshrc

Now all that we have to do is log out and back in:

# exit
# logout
> su -
# /usr/local/temp/bin/screen

Let’s take a look if the new compiler responds to cc (and fix c++ support along the way):

# ln -sf /usr/local/pkgsrc/gcc44/lib/libgcc_s.so.1 /usr/lib/libgcc_s.so.1
# cc -v
[...]
gcc version 4.4.7 (GCC)

GCC 4.4.7 running on FreeBSD 4.11

Yes, we really have GCC 4.4 running on FreeBSD 4.11! While it’s certainly not a modern compiler, it’s recent enough to build a lot of software. The latest release of OpenBSD, version 6.0 released on September 2016, still comes with GCC 4.2, BTW! Yes, OpenBSD maintained that all the time and heavily patch it. Still we now actually have a compiler available on FreeBSD 4.11 from 2005 which is two major versions newer!

With this we’re kind of back in business. But this post is already becoming quite long and for that reason I’m putting the “grand finale” off to one more post. See you there for the final outcome of this “little” experiment (which I hadn’t intended to write more than three posts for, but there you have it).

Updating FreeBSD 4.11 (2/4) – Digging up old graves

In part one I wrote about Legacy systems in general and showed a FreeBSD 4.11 installation for those of my readers who are interested in software history.

This post is about the first part of updating this fresh 4.11 system to a state that’s a bit less catastrophic. Remember: FreeBSD 4.11 was released in 2005 – however the ABI of each release is carved in stone with a .0 release. Which means that the software in the base system is from 4.0 and thus we venture back into the last millenium: 1999!

Initial state

To give you an idea what this means, here are a few program versions:

Various program’s versions in 4.11’s base system

So we have these programs among others:

GCC 2.95.4
Binutils 2.12.1
Perl 5.0
OpenSSH 3.5

To make matters worse, the ports tree for FreeBSD 4.11 is pretty dead, too. It’s important to get newer compilers running, but around 2005 FreeBSD used special releases to build GCC from (“gcc-core”) and I was not able to find a single mirror on the net that still holds those old and exotic files! Out of luck here. We’ll have to do without those ports.

“Modernizing” this is going to be interesting… Considering how fast the IT world moves, all of this is just as dead as it gets. So let’s prepare for the (code) smell and start digging up an old grave, shall we?

Allowing remote connection

After a passwordless login as root it makes sense to set up the right keymap (if you don’t use the default, that is). I had no idea how to do it on 4.11 and so I just gave the usual way of doing it a try – and was met with success:

# kbdmap

Looks like the keymap selection has not changed in all the time! Let’s try to make it persistent:

# echo keymap=german.iso.kbd >> /etc/rc.conf

I tried it out and it did just what I wanted. Time to try to add a regular user:

# adduser

The script is a bit different from what we’re used to today but in the end it does the same thing: Allow us to create a user. It’s important to add this user to the wheel group so that we’re able to su to root. However we need to give root a password first:

The useradd script in FreeBSD 4.11

# passwd

Ok, sshd should be running. Let’s check just to be sure:

# ps aux | grep sshd
[...]
root        75  0.0  0.1  2600 2040  ??  Is    6:26AM   0:00.11 /usr/sbin/sshd

Looks good. What about connectivity? Let’s see:

# ifconfig
vr0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        ether 00:05:5d:96:fa:f9
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
[...]

Nope, no connection. Luckily I paid attention when I the installer started from the CD and somewhere the strange-looking path of /stand caught my eye. Doing a little research, I found out that this directory used to be part of the OS and was removed in early FreeBSD 5 as it was mostly redundant with /rescue. What was in there, you ask? Have a look yourself:

# ls /stand
-sh             etc             minigzip        rm              tunefs
[               find            mount_mfs       route           usbd
arp             fsck            mount_nfs       rtsol           usbdevs
boot_crunch     gunzip          newfs           sed             zcat
camcontrol      gzip            pccardc         sh
cpio            help            pccardd         slattach
dhclient        hostname        ppp             sysinstall
dhclient-script ifconfig        pwd             test

Network configuration

There’s our friend sysinstall. I already said that it does more than just install the system. So let’s bring it up now:

# /stand/sysinstall

Sysinstall to configure the network

There we choose Networking -> Interfaces -> the appropriate NIC. No, I don’t want IPv6 and yes, DHCP is the thing.

Interface configuration using sysinstall

I’ve called the system vierelf which would be “foureleven” in English because I couldn’t think of anything better. And it’s just a test system anyway. Does the connection work now?

# ping elderlinux.org
PING elderlinux.org (212.77.232.71): 56 data bytes
64 bytes from 212.77.232.71: icmp_seq=0 ttl=54 time=24.993 ms

A little housekeeping

Alright! Let’s just take a look what services are listening right now:

# sockstat -4 -l
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      
root     dhclient   281    7 udp4   *:68                  *:*                  
root     sendmail    84    4 tcp4   *:25                  *:*                  
root     sendmail    84    6 tcp4   *:587                 *:*                  
root     sshd        75    4 tcp4   *:22                  *:*                  
root     syslogd     61    5 udp4   *:514                 *:*

Ugh! Time to make a few changes in /etc/rc.conf to deactivate any daemons except for SSH (which we need). This may not be strictly necessary but we want to improve the security of this system, right? And I wouldn’t trust those crusty old daemons at all. And whatever is not running won’t cause us any problems. So let’s get rid of them with extreme prejudice!

# vi /etc/rc.conf

To do so, add daemonname_enable=”NO” for each daemon and in case of sendmail use sendmail_enable=”NONE”.

FreeBSD 4.11 rc.conf with most daemons disabled

If you reboot now, sendmail and syslogd as well as cron, usbd and inetd will be disabled. That’s a starting point in securing the system. Let’s move on.

Replacing a dead tree

I’ll connect to the 4.11 box remotely over SSH because it’s much more convenient to have my trusty terminal at hand and to be able to copy and paste stuff:

% ssh kraileth@192.168.1.5
Unable to negotiate with 192.168.1.5 port 22: no matching host key type found. Their offer: ssh-dss

Ouch! I cannot even SSH into the system because its version of OpenSSH is so old that it only offers ssh-dss keys which have been deprecated for quite a while and disabled by default in OpenSSH >=7.0. So to connect to that old server, I have to tell my SSH client to accept ssh-dss for this connection:

% ssh kraileth@192.168.1.5 -oHostKeyAlgorithms=+ssh-dss

SSH login using “-oHostKeyAlgorithms=+ssh-dss”

Ok, I’m in. But what now? We cannot use FreeBSD’s ports tree but I strongly prefer some means of package management over installing stuff using make install. So how do we accomplish this? Enter pkgsrc. Pkgsrc is basically NetBSD’s fork of the FreeBSD ports tree. Being a NetBSD project however, it’s not limited to just NetBSD. It’s a truely portable way of building and managing software (I might write a separate post about it some time).

There’s just one problem: Downloading and decompressing the latest pkgsrc release (currently 2016Q4) won’t complete the bootstrapping process. Obviously FreeBSD 4.11 is no longer supported – which is not so much of a surprise. Time to try out older releases! After doing so I found out that 2009Q4 seems to be the last release to bootstrap successfully.

But here’s another problem: Pkgsrc doesn’t seem to keep older releases around and I also haven’t found them mirrored anywhere on the net. Pkgsrc uses CVS, however. So it’s possible to checkout older versions. FreeBSD comes with CVS as part of the base system. Unfortunately CVS works over SSH. And NetBSD’s CVS server won’t accept ssh-dss (which totally makes sense)! Since we don’t control the server, there’s also no way to just add a parameter or something to make it work. It simply doesn’t work that way.

Time to get CVS on my slightly more modern FreeBSD 11, do the checkout there and tar it all up to copy it over via scp! We’re going to get 2007Q2 instead, though, since we need things that won’t work on FreeBSD 4.11 with later versions. Oh, and if you’re not familiar with CVS, don’t worry. You don’t need to know what modules or tags are. Just copy the commands that I prepared for you and you’re good to go:

% sudo pkg install cvs
% cvs -danoncvs@anoncvs.netbsd.org:/cvsroot get -rpkgsrc-2007Q2 -P pkgsrc
% tar cvjf pkgsrc2007Q2.tbz2 pkgsrc/*
% scp pkgsrc2007Q2.tbz2 kraileth@192.168.1.5:/usr/home/kraileth

Then back on vierelf the next step is to prepare some directories and extract the pkgsrc tarball:

# mkdir -p /usr/local/temp /usr/pkgsrc
# cd /usr/pkgsrc
# mv /usr/home/kraileth/pkgsrc2007Q2.tbz2 .
# tar xvjf pkgsrc2007Q2
# rm pkgsrc2007Q2.tbz2
# mv pkgsrc 07

Bridgehead in hostile territory

Now we can bootstrap pkgsrc:

# cd /usr/pkgsrc/07/bootstrap
# ./bootstrap --prefix=/usr/local/temp --varbase=/usr/local/temp --pkgdbdir=/usr/local/temp/db

Pkgsrc 2007Q2 bootstrap complete!

Pkgsrc has been bootstrapped successfully! We just need to adjust the path variable so that the system picks up binaries from the new paths (and make those take precedence over the old system binaries). We could just change the PATH variable but it’s better to make the changes persistent. So let’s add the two new paths in the shell’s rc file in front of the others:

# vi /root/.cshrc

This is what needs to be prepended:
/usr/local/temp/sbin /usr/local/temp/bin

Root’s .cshrc file for the first phase pkgsrc

Now simply log out and become root again to have the new environment:

# logout
> su -

As a last step check if everything is right and we can access binaries in both paths:

# which bmake
/usr/local/temp/bin/bmake
# pkg_info 
bootstrap-mk-files-20061111 *.mk files for the bootstrap bmake utility
bmake-20051105nb3   Portable (autoconf) version of NetBSD 'make' utility
tnftp-20050625nb1   The enhanced FTP client in NetBSD
mtree-20070710      Utility for mapping and checking directory hierarchies
pax-20060202nb1     POSIX standard archiver with many extensions
pkg_install-20070710 Package management and administration tools for pkgsrc

Excellent! Now we have a working replacement for FreeBSD’s dead ports tree. This is definitely something that we can build upon.

Bring on some stability!

Lame pun, I know. Nevertheless it makes sense to… err… update the OS to the latest version. This is what we currently have:

# uname -a
FreeBSD vierelf.localdomain 4.11-RELEASE FreeBSD 4.11-RELEASE #0: Fri Jan 21 17:21:22 GMT 2005     root@perseus.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386

We’re on 4.11-RELEASE. The latest code for each release cycle is always in the -STABLE branch. Believe it or not: The latest code change was in April of 2014! The traditional way of getting FreeBSD code was over CVS. No, this time the problem is not ssh-dss. FreeBSD migrated from CVS to SVN (subversion) in 2008. FreeBSD CVS servers have been removed years ago. Therefore the old tools like cvsup are useless. Subversion is needed to checkout the source.

Luckily we have our pkgsrc ready. This old release has a very old port for subversion but that’s fair enough. There are a few source tarballs that are no longer available from the mirrors that pkgsrc knew for them. Not a big problem, we can download those manually:

# cd /usr/pkgsrc/07/distfiles
# fetch http://archive.apache.org/dist/httpd/httpd-2.0.61.tar.bz2
# fetch http://repository.timesys.com/buildsources/e/expat/expat-2.0.1/expat-2.0.1.tar.gz
# fetch http://download.nust.na/pub2/openpkg1/sources/DST/pkgconfig/pkg-config-0.21.tar.gz

Now we can build subversion:

# cd /usr/pkgsrc/07/devel/subversion-base
# bmake install clean clean-depends

Various dependencies will be downloaded, built and installed. Eventually subversion will be installed and available on the system. Time to tell the shell to look for new binaries and then checkout the stable source code:

# rehash
# svn co svn://svn.freebsd.org/base/stable/4 /usr/src

SVN checkout of the 4.11-STABLE code

The FreeBSD 4 base system never knew anything beyond CVS and cannot cope with the .svn directories that the svn checkout creates. World builds but the installation fails like this:

install: /usr/libdata/perl/5.00503/./.svn/text-base/Cwd.pm.svn-base: No such file or directory

Therefore it’s necessary to get rid of those disruptive directories:

# cd /usr/src
# find . -iname '.svn' -exec rm -rf {} \;

Now we can build world and kernel, install both and reboot the system:

# make buildworld
# make buildkernel
# make installkernel
# make installworld
# shutdown -r now

When the system comes back up we can SSH into it again. And there we can see that we’re on -STABLE now!

The newly built FreeBSD 4.11-STABLE

# uname -a
FreeBSD vierelf.localdomain 4.11-STABLE FreeBSD 4.11-STABLE #0: Sat Jan 28 11:53:06 GMT 2017     root@vierelf.localdomain:/usr/obj/usr/src/sys/GENERIC  i386

That’s it for today. We’re not quite there yet, but we’ve laid the groundwork for many more updates to come. Those will be described in the coming two posts of this mini series.