Updating FreeBSD 4.11 (4/4) – Reflecting radical resurrection

In the first post of this mini series I wrote about legacy systems and installing FreeBSD 4.11. The second one shows how to configure the fresh system for remote access, bootstrap Pkgsrc, install Subversion to checkout FreeBSD code and update the system to the stable branch. And part three mainly deals with upgrading OpenSSH and the compilers. This post details some more updates until we reach the final state that’s possible with such an old system (without resorting to extreme means).

Planting a new tree

So far we’ve built some packages from 2013 and before. Using a current pkgsrc tree won’t work – the various pkgsrc tools that our system has are too old. It might not be too big a step but we can use a tree from the second half of 2014. Of course the newer SSH that we built before is not currently in the path so we need to create a temporary symlink before we can use CVS again:

# ln -s /usr/local/temp/bin/ssh /usr/local/pkgsrc/bin/ssh
# rehash

# cd /usr/pkgsrc
# cvs -danoncvs@anoncvs.netbsd.org:/cvsroot get -rpkgsrc-2014Q3 -P pkgsrc
# mv pkgsrc 14
# rm /usr/local/pkgsrc/bin/ssh

What the system looks like package-wise at the beginning of part 4

Most of pkgsrc’s tools make use of NetBSD’s compatibility library. Unfortunately the version that comes with the new pkgsrc tree won’t build anymore on an OS as old as FreeBSD 4.11. Same thing for libfetch. But the newer tools will work with older versions of that libs, too. So let’s prepare those two – libfetch need’s some more love to build:

# cd /usr/pkgsrc/13/pkgtools/libnbcompat
# bmake

# cd /usr/pkgsrc/13/net/libfetch
# cp Makefile Makefile.bak
# sed '14i\\
CFLAGS=         -Wno-error' Makefile.bak > Makefile
# bmake

As a next step we’re going to do two updates. Yes, in theory we could use “bmake update” to update packages. We will not do that. The reason is that we needed to abuse pkgsrc quite a bit so far by mixing package versions from various trees. Since “bmake update” is a destructive command (it will happily uninstall programs as well as packages depending on them!) this can lead to all sort of fun things like unresolvable dependencies and such.

If you like pain, go ahead. I’ve been there and I can confirm that it does work for some packages. For a lot of them actually. But in those cases where it doesn’t, it tends to do so much damage that you’re better off starting over than trying to fix things… That’s why I’ll show you a safer method instead: Build a package and update via pkg_add! Also it really starts to show how old the system is that we’re trying to build rather new packages on. More and more of them require some trickery to persuade them to build – but hey, we’re doing a gross thing here, anyway. So there’s no real reason to complain!

# cd /usr/pkgsrc/14/pkgtools/pkg_install
# bmake extract
# rm -r work/libnbcompat/*
# rm -r work/libfetch/*
# cp -R /usr/pkgsrc/13/pkgtools/libnbcompat/work/libnbcompat-20120702/* /usr/pkgsrc/14/pkgtools/pkg_install/work/libnbcompat/
# cp -R /usr/pkgsrc/13/net/libfetch/work/libfetch-2.34/* /usr/pkgsrc/14/pkgtools/pkg_install/work/libfetch/
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/14/packages/All/pkg_install-20130902nb1.tgz

# cd /usr/pkgsrc/14/pkgtools/bootstrap-mk-files
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/14/packages/All/bootstrap-mk-files-20140516.tgz

We made it so far, now let’s make a daring move and just download the latest stable pkgsrc tree – released in January 2017:

# cd /usr/pkgsrc
# fetch http://cdn.netbsd.org/pub/pkgsrc/stable/pkgsrc-2016Q4.tar.bz2
# tar xvjf pkgsrc-2016Q4.tar.bz2
# rm pkgsrc-2016Q4.tar.bz2
# mv pkgsrc 16

Updating pkgsrc tools

Since mid 2014, pkgsrc makes use of a new package, cwrappers. During my test run I somehow managed to just get this package built. Despite taking notes I have no idea what I did to just make it work! It must have been something that looked like a dead end (which is why I didn’t include it in my notes) but somehow provided “getline”… I tried to get it working again for almost one whole Sunday but for the life of me couldn’t find out what I previously did… In the end I gave up and tried to find another solution. I found one but while it is way more complex it at least means that I got rid of that nasty blocker again:

# cd /usr/pkgsrc/16/pkgtools/cwrappers
# bmake extract
# rm -r work/libnbcompat/*
# cp -R /usr/pkgsrc/13/pkgtools/libnbcompat/work/libnbcompat-20120702/* /usr/pkgsrc/16/pkgtools/cwrappers/work/libnbcompat/
# cp work/cwrappers-20161125/mi_vector_hash.c work/cwrappers-20161125/mi_vector_hash.c.bak
# cp work/cwrappers-20161125/fixup-libtool.c work/cwrappers-20161125/fixup-libtool.c.bak
# sed 's/stdint.h/inttypes.h/' work/cwrappers-20161125/mi_vector_hash.c.bak > work/cwrappers-20161125/mi_vector_hash.c
# sed 's/stdint.h/inttypes.h/' work/cwrappers-20161125/fixup-libtool.c.bak > work/cwrappers-20161125/fixup-libtool.c
# cp /usr/pkgsrc/14/pkgtools/cwrappers/files/bin/getline.c /usr/pkgsrc/16/pkgtools/cwrappers/work/cwrappers-20161125/getline.c.bak
# sed 's/ssize_t/size_t/' work/cwrappers-20161125/getline.c.bak > work/cwrappers-20161125/getline.c
# cp work/cwrappers-20161125/common.h work/cwrappers-20161125/common.h.bak
# sed '107i\\
size_t  getline(char **, size_t *, FILE *);' work/cwrappers-20161125/common.h.bak > work/cwrappers-20161125/common.h
# cp work/cwrappers-20161125/Makefile work/cwrappers-20161125/Makefile.bak
# sed '14i\\
LIB_SRCS+=      getline.c' work/cwrappers-20161125/Makefile.bak > work/cwrappers-20161125/Makefile
# bmake install clean clean-depends

Phew! Fortunately the next few updates are straight forward:

# cd /usr/pkgsrc/16/pkgtools/bootstrap-mk-files
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/bootstrap-mk-files-20160908.tgz

# cd /usr/pkgsrc/16/devel/bmake
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/bmake-20150505.tgz

# cd /usr/pkgsrc/16/net/tnftp
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/tnftp-20151004nb1.tgz

Next is another one that requires some patching:

# cd /usr/pkgsrc/16/pkgtools/digest/
# bmake extract
# cp work/digest-20160304/sha3.h work/digest-20160304/sha3.h.bak
# cp work/digest-20160304/keccak.c work/digest-20160304/keccak.c.bak
# cp work/digest-20160304/keccak.h work/digest-20160304/keccak.h.bak
# cp work/digest-20160304/sha3.c work/digest-20160304/sha3.c.bak
# sed 's/stdint.h/inttypes.h/' work/digest-20160304/sha3.h.bak > work/digest-20160304/sha3.h
# sed 's/stdint.h/inttypes.h/' work/digest-20160304/keccak.c.bak > work/digest-20160304/keccak.c
# sed 's/stdint.h/inttypes.h/' work/digest-20160304/keccak.h.bak > work/digest-20160304/keccak.h
# sed 's/stdint.h/inttypes.h/' work/digest-20160304/sha3.c.bak > work/digest-20160304/sha3.c
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/digest-20160304.tgz

Updating installed packages

Let’s update gettext first as a lot of packages need that one; xz is one of the packages that is linked against the old one and since libintl received a soname bump, it needs to be rebuilt. Since we want to update it anyway that’s not too bad. But there are other packages that we cannot update which depend on the old lib. So we’ll have to create a symlink to satisfy their need, too:

# cd /usr/pkgsrc/16/devel/gettext-lib
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gettext-lib-0.19.8.1.tgz

# cd /usr/pkgsrc/16/archivers/xz
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/xz-5.2.2.tgz

# ln -s /usr/local/pkgsrc/lib/libintl.so.9 /usr/local/pkgsrc/lib/libintl.so.7
# cd /usr/pkgsrc/16/devel/gettext-tools
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gettext-tools-0.19.8.1.tgz

Next in line is some more typical build dependencies:

# cd /usr/pkgsrc/16/devel/libtool-base
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/libtool-base-2.4.2nb13.tgz

# cd /usr/pkgsrc/16/devel/m4
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/m4-1.4.17.tgz

# cd /usr/pkgsrc/16/devel/bison
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/bison-3.0.4nb3.tgz

Just a few more packages and we’ll have updated most packages that can be updated (a few like zip and nbpatch can’t):

# cd /usr/pkgsrc/16/shells/bash
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/bash-4.4.005.tgz

# cd /usr/pkgsrc/16/lang/perl5
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/perl-5.24.0.tgz

# cd /usr/pkgsrc/16/devel/autoconf
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/autoconf-2.69nb7.tgz

# cd /usr/pkgsrc/16/devel/gmake
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gmake-4.1nb3.tgz

Rebuilding the compiler

First we need to update the two math libraries (and create another symlink so we can go on compiling):

# cd /usr/pkgsrc/16/devel/gmp
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gmp-6.1.2.tgz
# ln -s /usr/local/pkgsrc/lib/libgmp.so.13 /usr/local/pkgsrc/lib/libgmp.so.11

# cd /usr/pkgsrc/16/math/mpfr
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/mpfr-3.1.5.tgz

This unfortunately breaks the compiler. But we can still resort to the old GCC3 to build GCC4 again, right? Right:

# cp /usr/pkgsrc/13/distfiles/gcc-4.4.7.tar.bz2 /usr/pkgsrc/16/distfiles
# cp /root/.cshrc /root/.cshrc.bak
# sed 's:pkgsrc/gcc44:temp/gcc34:' /root/.cshrc.bak > /root/.cshrc
# source /root/.cshrc
# cc -v

While we probably still don’t need Object-C or Java we could in fact build GCC with them this time. Java requires Python2.7 installed but that can actually be built from the 2014 tree! The problem is that building Java requires more RAM than is available on 32 bit machines and will for that reason fail. However Java is deactivated by default for GCC 4.4 in the 2016 tree. So let’s just get rid of our custom options, build the default package and set the correct path again:

# cp /usr/local/pkgsrc/etc/mk.conf /usr/local/pkgsrc/etc/mk.conf.bak
# sed '/PKG_OPTIONS.gcc44/d' /usr/local/pkgsrc/etc/mk.conf.bak > /usr/local/pkgsrc/etc/mk.conf
# cd /usr/pkgsrc/16/lang/gcc44
# bmake package clean clean-depends
# pkg_add -uu /usr/pkgsrc/16/packages/All/gcc44-4.4.7nb7.tgz

Almost everything updated!

Now we only have to restore the correct path and then we have the GCC4 back (with a newer patch level):

# cp /root/.cshrc.bak /root/.cshrc
# source /root/.cshrc
# cc -v
gcc version 4.4.7 (GCC)

Modern OpenSSH

There’s one more package to build that needs a bit of care: Pkgconf. It’s a simpler replacement for the older pkg-config but it won’t work out of the box for us:

# cd /usr/pkgsrc/16/devel/pkgconf
# bmake extract
# cp work/pkgconf-1.0.1/libpkgconf/stdinc.h work/pkgconf-1.0.1/libpkgconf/stdinc.h.bak
# cp work/pkgconf-1.0.1/getopt_long.h work/pkgconf-1.0.1/getopt_long.h.bak
# sed 's/stdint.h/inttypes.h/' work/pkgconf-1.0.1/libpkgconf/stdinc.h.bak > work/pkgconf-1.0.1/libpkgconf/stdinc.h
# sed 's/stdint.h/inttypes.h/' work/pkgconf-1.0.1/getopt_long.h.bak > work/pkgconf-1.0.1/getopt_long.h
# bmake install clean clean-depends

Finally the time has come to do what I wanted to do in the first place, provide a recent version of OpenSSH! Of course it’s also necessary to generate new host keys once more. And then, just to prove everything works when the machine boots, let’s just restart the machine after adjusting the sshd path:

# cd /usr/pkgsrc/16/security/openssh
# bmake install clean clean-depends
# rehash
# ssh -V

# ssh-keygen -f /usr/local/pkgsrc/etc/ssh/ssh_host_rsa_key -N '' -t rsa
# ssh-keygen -f /usr/local/pkgsrc/etc/ssh/ssh_host_dsa_key -N '' -t dsa
# mkdir -p /usr/local/pkgsrc/run

# cp /etc/rc.conf /etc/rc.conf.bak
# sed 's:temp/sbin:pkgsrc/sbin:' /etc/rc.conf.bak > /etc/rc.conf
# shutdown -r now

Generating new host keys for OpenSSH

Now we need to remove the vierelf entry in ~/.ssh/known_hosts before we connect again. Doing so in verbose mode even shows that the 4.11 box now has a newer version of OpenSSH installed that the FreeBSD 11 workstation that I use to connect to it! 😀

FreeBSD 4.11 running a newer OpenSSH than my FreeBSD 11.0 workstation!

Conclusion

FreeBSD 4.11 is really, really, really old now. But you can get surprisingly far in running somewhat modern software on it – more recent software at least than I initially thought would be possible! And you? What was your bet? Would you have guessed that I’d make it up to the 2016Q4 pkgsrc tree and even install the latest version of OpenSSL and OpenSSH?

Here’s a little summary of some important program updates:

binutils 2.12.1 (2002) -> binutils 2.17 (2006)
perl 5.005 (1998) -> perl 5.8 (2002) -> perl 5.18 (2013) -> perl 5.24 (2016)
GCC 2.95.4 (2001) -> GCC 3.4.6 (2006) -> GCC 4.4.7 (2012)
OpenSSH 3.5 (2002) -> OpenSSH 4.6 (2007) -> OpenSSH 7.3 (2016)

Not too bad, eh? The notable exception here is binutils. Newer versions would probably be possible but there’s a gap in pkgsrc – which stuck with 2.17 for a long time and then directly moved to 2.22 which no longer builds on FreeBSD 4.11. GCC 4.5.3 does build BTW but something goes sideways and the comparison of stage 2 and 3 fails for quite some files.

I’ve met my initial goal to provide a newer version of OpenSSH, surpassing all expectations that I had. There’s room for more of course but that’s not worth another post. I’m going to add sudo and since Python 2.7 can be built it might even be possible to manage the 4.11 servers using salt-ssh (the ordinary SaltStack doesn’t work as it requires ZeroMQ which looks like it cannot be built)! We have a recent version of bash and can thus do some pretty nifty things with the right .bashrc.

This whole adventure took far longer than I had anticipated – a bit over a month instead of the intended two weekends! But that was mostly because I decided to start over with a clean system several times to ensure that everything works as I wrote it down here (and because GCC4 simply takes so long to build on the only spare machine that I had for this…). But it has been an interesting ride and I don’t regret spending some time on the legendary FreeBSD 4.11!

Oh, and my special thanks to everybody involved with Pkgsrc! I usually don’t have much use for NetBSD but Pkgsrc is extremely useful. I might use it in the future on other systems (like Linux), too. And thanks to you for reading. I hope that you enjoyed it as well!

Updating FreeBSD 4.11 (3/4) – Neophyte’s notorious necromancy

The first post of this mini series was about legacy systems in general and about what installing the old FreeBSD 4.11 is like. In the second one I showed the initial configuration of the system, how to SSH into it despite the obsolete DSA host key and how to bootstrap pkgsrc, NetBSD’s portable ports tree. I also covered the installation of SVN, checking out of the 4.11-STABELE code and updating the system. This post will cover installing newer software.

Any bets?

So far we have a pkgsrc tree from mid 2007 and things seem to be working. However that’s pretty close to 4.11’s release in 2005 and thus not too amazing. Working with such an old system there are plenty of cases which mean “game over”. Here are just three errors of that kind which you can encounter trying to build more modern software:

/usr/libexec/elf/ld: cannot find -lpthread

There’s no modern pthreads available on 4.11. Game over.

/usr/include/sys/resource.h:58: error: field 'ru_utime' has incomplete type

We’ll have to do with very old system headers missing a lot of what we take for granted today. Game over again.

fileio.o(.text+0x354): undefined reference to `towupper'
collect2: ld returned 1 exit status

Sorry, that ancient libc that we have on our system doesn’t provide that symbol. Game over yet again.

How far do you think can we take it in building and installing more recent software? Make a guess now and see if you were right! To be honest I was not expecting the end result. Not at all. So let’s get back to work!

In for a screening

We’re going to compile a lot of stuff this time – building SVN and dependencies was just a warm up. And what do you do when you’re building stuff remotely over SSH? You’re doing so in a screen or tmux session of course. Neither is part of the base system so we’ve got to build one. Tmux was not yet available in 2007 so it’s not too hard a choice:

# cd /usr/pkgsrc/07/misc/screen
# bmake install clean clean-depends
# rehash
# screen

GNU screen started up and ready

If you don’t know screen do some reading because you will want to start using it (or rather the superior tmux). It basically allows you to detach from a session and reconnect later – and your programs will continue running on the remote system even while you’re logged out. You can also resume the session from another terminal or computer, share sessions, etc. And that’s just one of the things that it does. There are other features like allowing you to have multiple shell instances in just one terminal between which you can switch back and forth (think tabs of a browser) and a lot more. Should you not like this (what’s wrong with you?!), fine. Don’t install screen. It’s optional.

Replacing the front door lock

Now it’s time to take care of the main problem of our system: That darned version 3.5 of OpenSSH! Let’s build whatever our pkgsrc tree has to offer:

# cd /usr/pkgsrc/07/security/openssh
# bmake install clean clean-depends
# rehash
# ssh -V
OpenSSH_4.6p1, OpenSSL 0.9.7d-p1 17 Mar 2004

Still far from a modern version of OpenSSH but also a lot better. And the best thing: It supports RSA keys. Let’s generate host keys with this newer SSH and make it the version that FreeBSD launches during startup:

# ssh-keygen -f /usr/local/temp/etc/ssh/ssh_host_key -N '' -t rsa1
# ssh-keygen -f /usr/local/temp/etc/ssh/ssh_host_rsa_key -N '' -t rsa
# ssh-keygen -f /usr/local/temp/etc/ssh/ssh_host_dsa_key -N '' -t dsa
# mkdir -p /usr/local/temp/run
# echo 'sshd_program="/usr/local/temp/sbin/sshd"' >> /etc/rc.conf

Ok, everything is in place. We could reboot now – or just kill off the old daemon and launch the new one. Let’s first look for SSHD and see which PID it has (this of course varies from system to system!):

# ps aux | grep sshd

Replacing SSHD

Got it? Great, let’s kill it (your SSH connection is maintained by a child and it’s generally save to kill the parent. You won’t lose your SSH connection!), start the new one and ensure that it’s running:

# kill [PID on your system]
# /usr/local/temp/sbin/sshd
# ps aux | grep sshd

What’s this? It looks like it’s not running! Yes, it looks like it but actually it should be running… Let’s grep again:

# ps aux | grep local

This does return one process – and trust me it’s actually our new sshd. What’s happening here is this: The output of ps is truncated because more wouldn’t fit on the screen. And only that data is handed to grep! So the process with the name /usr/local/temp that we found (see the screenshot above) is actually /usr/local/temp/sbin/sshd with the last part of it cut off… This is why grep doesn’t find “sshd”. There’s a funny way to fix this, though: Maximize your terminal emulator so that more space is available. Then grep will find sshd!

Now we can quite the old SSH session so we can make one with the new server. We can even keep our screen session open, but we need to detach from it by pressing CTRL-A and then D before we logout from vierelf:

[detached]

# logout
> exit
Connection to 192.168.1.5 closed.

Time to edit your known hosts and get rid of the former host key for vierelf or else you’ll see that scary SSH warning when you try to login again. Oh, and you can leave out that compatibility option from now on – which is a major step ahead! When you’re back in, you can resume the screen session:

% ssh kraileth@192.168.1.5
> su -
# screen -r

Connecting to the new SSH server (debug mode)

Compiler: from antediluvian to ancient

Alright. Currently we have the last version of the second generation of GCC on our system. We totally need to get our hands on something newer. How about updating the last version of generation three? Let’s try that! We only want the C and C++ compilers. Fortran is deactivated by default for this version (it would need GMP installed and the version of GMP that’s in the tree requires GCC3. It’s a good idea to avoid that potential circular dependency). However Java and Object-C are activated. There’s no need to waste time on them, they should be deactivated as well. The following sed command may look a bit complex, but it’s not that bad. Just copy all three lines that make up that single command and you’re good to go:

# cd /usr/pkgsrc/07/lang/gcc34
# cp Makefile Makefile.bak
# sed -e '64,65d' -e '63a\\
BUILD_JAVA?=    NO' -e '63a\\
BUILD_OBJC?=    NO' Makefile.bak > Makefile
# bmake install clean clean-depends

After installing that newer GCC, the path needs to be changed again so that the system picks it up instead of the older system compiler:

# vi /root/.cshrc

Prepend the following path to the PATH variable:

/usr/local/temp/gcc34/bin

Now let’s log out and in again and see if the new compiler is available:

# exit
[screen is terminating]
# logout
> su -
# screen
# cc -v
[...]
gcc version 3.4.6

Updating pkgsrc

Since we also have a more recent OpenSSH now, we can checkout a newer copy of pkgsrc from CVS! That takes a while, be patient. Even after it is finished downloading (and you see no new lines on the screen) it will still take some time to clean things up. This is normal and you have to wait a little longer. Don’t CTRL+C it as that would leave your tree in bad shape!

# cd /usr/pkgsrc
# cvs -danoncvs@anoncvs.netbsd.org:/cvsroot get -rpkgsrc-2009Q4 -P pkgsrc
# mv pkgsrc 09

Thanks to the newer SSH: CVS works now, too!

We’ll need some ports from there later. But since we have GCC 3 available now we can also grab an even newer copy and primarily use that one:

# cvs -danoncvs@anoncvs.netbsd.org:/cvsroot get -rpkgsrc-2013Q2 -P pkgsrc
# mv pkgsrc 13

We’re going to start a fresh environment, using only GCC (and sshd) from the old one. To do so we first bootstrap the pkgsrc from 2013 into a new directory:

# mkdir /usr/local/pkgsrc
# cd /usr/pkgsrc/13/bootstrap
# ./bootstrap --prefix=/usr/local/pkgsrc --varbase=/usr/local/pkgsrc

The next step is to adjust the path variable so that the binaries from the new location are being used. To do so we need to replace /usr/local/temp with /usr/local/pkgsrc for both sbin and bin. Don’t change the compiler path, though! GCC 3 will remain in temp. After logging out and back in, screen is no longer in PATH so we need to execute it with the absolute path:

# cp /root/.cshrc /root/.cshrc.bak
# sed -e 's:temp/bin:pkgsrc/bin:' -e 's:temp/sbin:pkgsrc/sbin:' /root/.cshrc.bak > /root/.cshrc
# exit
# logout
> su -
# /usr/local/temp/bin/screen

Cherry-picking dependencies

This gives us a way to easily build software from 2013. Let’s continue on by fetching some source tarballs by hand that are no longer available on the mirrors that pkgsrc knew for them:

# cd /usr/pkgsrc/09/distfiles
# fetch http://ftp.cc.uoc.gr/mirrors/NetBSD/packages/distfiles/binutils-2.17.tar.gz
# fetch http://ftp.cc.uoc.gr/mirrors/NetBSD/packages/distfiles/pkg-config-0.23.tar.gz

The following part is not too interesting: We’re going to build the dependencies in preparation for the next big step. In general we try to build the newest version possible (2013) but resort to old (2009) or even older (2007) where necessary if newer versions don’t build for various reasons:

# cd /usr/pkgsrc/13/converters/libiconv
# bmake install clean clean-depends

Zip from 2009 and onwards is incompatible with FreeBSD 4.11’s libc. And the 2007 version expects tar in a location where there’s none on our system. Instead of building tar we can safely symlink it:

# ln -s /usr/bin/tar /usr/local/pkgsrc/bin/tar
# cd /usr/pkgsrc/07/archivers/zip
# bmake install clean clean-depends

The binutils are a special case. The port normally builds the programs of which it consists with a prefix so they don’t get in the way of the system binaries. Since we actually want to use them instead of the old stuff from the base system, we need to get rid of that prefix:

# cd /usr/pkgsrc/09/devel/binutils
# bmake GNU_PROGRAM_PREFIX='' install clean clean-depends
# rehash
# ld -v
GNU ld version 2.17

The next few are trivial:

# cd /usr/pkgsrc/09/devel/gettext-tools
# bmake install clean clean-depends

# cd /usr/pkgsrc/13/devel/m4
# bmake install clean clean-depends

# cd /usr/pkgsrc/09/devel/bison
# bmake install clean clean-depends

The bash port from 2013 would draw in a newer version of gettext which would not build. But bash can actually be built with the old one, too. So we have to make a simple change in the buildlink file for gettext in 2013’s pkgsrc tree:

# cd /usr/pkgsrc/13/devel/gettext-lib
# cp buildlink3.mk buildlink3.mk.bak
# sed 's/0.18/0.14/g' buildlink3.mk.bak > buildlink3.mk

With that change the next port can be built:

# cd /usr/pkgsrc/13/shells/bash
# bmake install clean clean-depends

Next in line is perl. The 2013 port would however build with dtrace support by default – which was of course not available on 4.11. Therefore it needs to be switched off by making an addition to the pkgsrc config file:

# vi /usr/local/pkgsrc/etc/mk.conf

Add the following line at the end of the file (but above .endif):

PKG_OPTIONS.perl=       -dtrace

Now let’s build the last few dependencies:

# cd /usr/pkgsrc/13/lang/perl5
# bmake install clean clean-depends

# cd /usr/pkgsrc/13/archivers/xz
# bmake install clean clean-depends

# cd /usr/pkgsrc/09/devel/autoconf
# bmake install clean clean-depends

Compiler: from ancient to old

With this all dependencies from earlier than 2013 in place we are good to go for the biggest update. We’re still not interested in Java and Object-C, so let’s edit pkgsrc’s configuration again:

# vi /usr/local/pkgsrc/etc/mk.conf

and add one more line (e.g. after the perl one):

PKG_OPTIONS.gcc44=      -gcc-java -gcc-objc

Building the newer version of GCC means building two more dependencies as well, one of which is libgmp. GMP is the first package so far that uses C++ and in fact our C++ compiler has been broken the whole time. Luckily a symlink can heal it and another one will make GCC happy so that we can finally build it – which takes quite a bit of time (I’ve seen the compilation stop at one point and I’m not sure what happens there. But just calling bmake again will eventually complete the build process!):

# ln -s /usr/local/pkgsrc/lib/libiconv.so.7 /usr/lib/libiconv.so.7
# ln -s /usr/local/temp/gcc34/lib/libgcc_s.so.1 /usr/lib/libgcc_s.so.1
# cd /usr/pkgsrc/13/lang/gcc44/
# bmake install clean clean-depends

Once it’s build, we need to change our PATH so that the newer GCC is the primary compiler:

# mv /root/.cshrc /root/.cshrc.bak
# sed 's:temp/gcc34:pkgsrc/gcc44:' /root/.cshrc.bak > /root/.cshrc

Now all that we have to do is log out and back in:

# exit
# logout
> su -
# /usr/local/temp/bin/screen

Let’s take a look if the new compiler responds to cc (and fix c++ support along the way):

# ln -sf /usr/local/pkgsrc/gcc44/lib/libgcc_s.so.1 /usr/lib/libgcc_s.so.1
# cc -v
[...]
gcc version 4.4.7 (GCC)

GCC 4.4.7 running on FreeBSD 4.11

Yes, we really have GCC 4.4 running on FreeBSD 4.11! While it’s certainly not a modern compiler, it’s recent enough to build a lot of software. The latest release of OpenBSD, version 6.0 released on September 2016, still comes with GCC 4.2, BTW! Yes, OpenBSD maintained that all the time and heavily patch it. Still we now actually have a compiler available on FreeBSD 4.11 from 2005 which is two major versions newer!

With this we’re kind of back in business. But this post is already becoming quite long and for that reason I’m putting the “grand finale” off to one more post. See you there for the final outcome of this “little” experiment (which I hadn’t intended to write more than three posts for, but there you have it).

Updating FreeBSD 4.11 (2/4) – Digging up old graves

In part one I wrote about Legacy systems in general and showed a FreeBSD 4.11 installation for those of my readers who are interested in software history.

This post is about the first part of updating this fresh 4.11 system to a state that’s a bit less catastrophic. Remember: FreeBSD 4.11 was released in 2005 – however the ABI of each release is carved in stone with a .0 release. Which means that the software in the base system is from 4.0 and thus we venture back into the last millenium: 1999!

Initial state

To give you an idea what this means, here are a few program versions:

Various program’s versions in 4.11’s base system

So we have these programs among others:

GCC 2.95.4
Binutils 2.12.1
Perl 5.0
OpenSSH 3.5

To make matters worse, the ports tree for FreeBSD 4.11 is pretty dead, too. It’s important to get newer compilers running, but around 2005 FreeBSD used special releases to build GCC from (“gcc-core”) and I was not able to find a single mirror on the net that still holds those old and exotic files! Out of luck here. We’ll have to do without those ports.

“Modernizing” this is going to be interesting… Considering how fast the IT world moves, all of this is just as dead as it gets. So let’s prepare for the (code) smell and start digging up an old grave, shall we?

Allowing remote connection

After a passwordless login as root it makes sense to set up the right keymap (if you don’t use the default, that is). I had no idea how to do it on 4.11 and so I just gave the usual way of doing it a try – and was met with success:

# kbdmap

Looks like the keymap selection has not changed in all the time! Let’s try to make it persistent:

# echo keymap=german.iso.kbd >> /etc/rc.conf

I tried it out and it did just what I wanted. Time to try to add a regular user:

# adduser

The script is a bit different from what we’re used to today but in the end it does the same thing: Allow us to create a user. It’s important to add this user to the wheel group so that we’re able to su to root. However we need to give root a password first:

The useradd script in FreeBSD 4.11

# passwd

Ok, sshd should be running. Let’s check just to be sure:

# ps aux | grep sshd
[...]
root        75  0.0  0.1  2600 2040  ??  Is    6:26AM   0:00.11 /usr/sbin/sshd

Looks good. What about connectivity? Let’s see:

# ifconfig
vr0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        ether 00:05:5d:96:fa:f9
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
[...]

Nope, no connection. Luckily I paid attention when I the installer started from the CD and somewhere the strange-looking path of /stand caught my eye. Doing a little research, I found out that this directory used to be part of the OS and was removed in early FreeBSD 5 as it was mostly redundant with /rescue. What was in there, you ask? Have a look yourself:

# ls /stand
-sh             etc             minigzip        rm              tunefs
[               find            mount_mfs       route           usbd
arp             fsck            mount_nfs       rtsol           usbdevs
boot_crunch     gunzip          newfs           sed             zcat
camcontrol      gzip            pccardc         sh
cpio            help            pccardd         slattach
dhclient        hostname        ppp             sysinstall
dhclient-script ifconfig        pwd             test

Network configuration

There’s our friend sysinstall. I already said that it does more than just install the system. So let’s bring it up now:

# /stand/sysinstall

Sysinstall to configure the network

There we choose Networking -> Interfaces -> the appropriate NIC. No, I don’t want IPv6 and yes, DHCP is the thing.

Interface configuration using sysinstall

I’ve called the system vierelf which would be “foureleven” in English because I couldn’t think of anything better. And it’s just a test system anyway. Does the connection work now?

# ping elderlinux.org
PING elderlinux.org (212.77.232.71): 56 data bytes
64 bytes from 212.77.232.71: icmp_seq=0 ttl=54 time=24.993 ms

A little housekeeping

Alright! Let’s just take a look what services are listening right now:

# sockstat -4 -l
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      
root     dhclient   281    7 udp4   *:68                  *:*                  
root     sendmail    84    4 tcp4   *:25                  *:*                  
root     sendmail    84    6 tcp4   *:587                 *:*                  
root     sshd        75    4 tcp4   *:22                  *:*                  
root     syslogd     61    5 udp4   *:514                 *:*

Ugh! Time to make a few changes in /etc/rc.conf to deactivate any daemons except for SSH (which we need). This may not be strictly necessary but we want to improve the security of this system, right? And I wouldn’t trust those crusty old daemons at all. And whatever is not running won’t cause us any problems. So let’s get rid of them with extreme prejudice!

# vi /etc/rc.conf

To do so, add daemonname_enable=”NO” for each daemon and in case of sendmail use sendmail_enable=”NONE”.

FreeBSD 4.11 rc.conf with most daemons disabled

If you reboot now, sendmail and syslogd as well as cron, usbd and inetd will be disabled. That’s a starting point in securing the system. Let’s move on.

Replacing a dead tree

I’ll connect to the 4.11 box remotely over SSH because it’s much more convenient to have my trusty terminal at hand and to be able to copy and paste stuff:

% ssh kraileth@192.168.1.5
Unable to negotiate with 192.168.1.5 port 22: no matching host key type found. Their offer: ssh-dss

Ouch! I cannot even SSH into the system because its version of OpenSSH is so old that it only offers ssh-dss keys which have been deprecated for quite a while and disabled by default in OpenSSH >=7.0. So to connect to that old server, I have to tell my SSH client to accept ssh-dss for this connection:

% ssh kraileth@192.168.1.5 -oHostKeyAlgorithms=+ssh-dss

SSH login using “-oHostKeyAlgorithms=+ssh-dss”

Ok, I’m in. But what now? We cannot use FreeBSD’s ports tree but I strongly prefer some means of package management over installing stuff using make install. So how do we accomplish this? Enter pkgsrc. Pkgsrc is basically NetBSD’s fork of the FreeBSD ports tree. Being a NetBSD project however, it’s not limited to just NetBSD. It’s a truely portable way of building and managing software (I might write a separate post about it some time).

There’s just one problem: Downloading and decompressing the latest pkgsrc release (currently 2016Q4) won’t complete the bootstrapping process. Obviously FreeBSD 4.11 is no longer supported – which is not so much of a surprise. Time to try out older releases! After doing so I found out that 2009Q4 seems to be the last release to bootstrap successfully.

But here’s another problem: Pkgsrc doesn’t seem to keep older releases around and I also haven’t found them mirrored anywhere on the net. Pkgsrc uses CVS, however. So it’s possible to checkout older versions. FreeBSD comes with CVS as part of the base system. Unfortunately CVS works over SSH. And NetBSD’s CVS server won’t accept ssh-dss (which totally makes sense)! Since we don’t control the server, there’s also no way to just add a parameter or something to make it work. It simply doesn’t work that way.

Time to get CVS on my slightly more modern FreeBSD 11, do the checkout there and tar it all up to copy it over via scp! We’re going to get 2007Q2 instead, though, since we need things that won’t work on FreeBSD 4.11 with later versions. Oh, and if you’re not familiar with CVS, don’t worry. You don’t need to know what modules or tags are. Just copy the commands that I prepared for you and you’re good to go:

% sudo pkg install cvs
% cvs -danoncvs@anoncvs.netbsd.org:/cvsroot get -rpkgsrc-2007Q2 -P pkgsrc
% tar cvjf pkgsrc2007Q2.tbz2 pkgsrc/*
% scp pkgsrc2007Q2.tbz2 kraileth@192.168.1.5:/usr/home/kraileth

Then back on vierelf the next step is to prepare some directories and extract the pkgsrc tarball:

# mkdir -p /usr/local/temp /usr/pkgsrc
# cd /usr/pkgsrc
# mv /usr/home/kraileth/pkgsrc2007Q2.tbz2 .
# tar xvjf pkgsrc2007Q2
# rm pkgsrc2007Q2.tbz2
# mv pkgsrc 07

Bridgehead in hostile territory

Now we can bootstrap pkgsrc:

# cd /usr/pkgsrc/07/bootstrap
# ./bootstrap --prefix=/usr/local/temp --varbase=/usr/local/temp --pkgdbdir=/usr/local/temp/db

Pkgsrc 2007Q2 bootstrap complete!

Pkgsrc has been bootstrapped successfully! We just need to adjust the path variable so that the system picks up binaries from the new paths (and make those take precedence over the old system binaries). We could just change the PATH variable but it’s better to make the changes persistent. So let’s add the two new paths in the shell’s rc file in front of the others:

# vi /root/.cshrc

This is what needs to be prepended:
/usr/local/temp/sbin /usr/local/temp/bin

Root’s .cshrc file for the first phase pkgsrc

Now simply log out and become root again to have the new environment:

# logout
> su -

As a last step check if everything is right and we can access binaries in both paths:

# which bmake
/usr/local/temp/bin/bmake

# pkg_info 
bootstrap-mk-files-20061111 *.mk files for the bootstrap bmake utility
bmake-20051105nb3   Portable (autoconf) version of NetBSD 'make' utility
tnftp-20050625nb1   The enhanced FTP client in NetBSD
mtree-20070710      Utility for mapping and checking directory hierarchies
pax-20060202nb1     POSIX standard archiver with many extensions
pkg_install-20070710 Package management and administration tools for pkgsrc

Excellent! Now we have a working replacement for FreeBSD’s dead ports tree. This is definitely something that we can build upon.

Bring on some stability!

Lame pun, I know. Nevertheless it makes sense to… err… update the OS to the latest version. This is what we currently have:

# uname -a
FreeBSD vierelf.localdomain 4.11-RELEASE FreeBSD 4.11-RELEASE #0: Fri Jan 21 17:21:22 GMT 2005     root@perseus.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386

We’re on 4.11-RELEASE. The latest code for each release cycle is always in the -STABLE branch. Believe it or not: The latest code change was in April of 2014! The traditional way of getting FreeBSD code was over CVS. No, this time the problem is not ssh-dss. FreeBSD migrated from CVS to SVN (subversion) in 2008. FreeBSD CVS servers have been removed years ago. Therefore the old tools like cvsup are useless. Subversion is needed to checkout the source.

Luckily we have our pkgsrc ready. This old release has a very old port for subversion but that’s fair enough. There are a few source tarballs that are no longer available from the mirrors that pkgsrc knew for them. Not a big problem, we can download those manually:

# cd /usr/pkgsrc/07/distfiles
# fetch http://archive.apache.org/dist/httpd/httpd-2.0.61.tar.bz2
# fetch http://repository.timesys.com/buildsources/e/expat/expat-2.0.1/expat-2.0.1.tar.gz
# fetch http://download.nust.na/pub2/openpkg1/sources/DST/pkgconfig/pkg-config-0.21.tar.gz

Now we can build subversion:

# cd /usr/pkgsrc/07/devel/subversion-base
# bmake install clean clean-depends

Various dependencies will be downloaded, built and installed. Eventually subversion will be installed and available on the system. Time to tell the shell to look for new binaries and then checkout the stable source code:

# rehash
# svn co svn://svn.freebsd.org/base/stable/4 /usr/src

SVN checkout of the 4.11-STABLE code

The FreeBSD 4 base system never knew anything beyond CVS and cannot cope with the .svn directories that the svn checkout creates. World builds but the installation fails like this:

install: /usr/libdata/perl/5.00503/./.svn/text-base/Cwd.pm.svn-base: No such file or directory

Therefore it’s necessary to get rid of those disruptive directories:

# cd /usr/src
# find . -iname '.svn' -exec rm -rf {} \;

Now we can build world and kernel, install both and reboot the system:

# make buildworld
# make buildkernel
# make installkernel
# make installworld
# shutdown -r now

When the system comes back up we can SSH into it again. And there we can see that we’re on -STABLE now!

The newly built FreeBSD 4.11-STABLE

# uname -a
FreeBSD vierelf.localdomain 4.11-STABLE FreeBSD 4.11-STABLE #0: Sat Jan 28 11:53:06 GMT 2017     root@vierelf.localdomain:/usr/obj/usr/src/sys/GENERIC  i386

That’s it for today. We’re not quite there yet, but we’ve laid the groundwork for many more updates to come. Those will be described in the coming two posts of this mini series.

Updating FreeBSD 4.11 (1/4) – Blast from the past

Ah, weekend. And it’s a nice day, too: The sun shines on the snowy landscape! A perfect day to go out with the children – or to embark on a vastly different kind of adventure… Yes, you read that title right. This is not about four FreeBSD 11 machines. It’s about one FreeBSD 4.11 machine in 2017!

Legacy systems in general

Why even bother with FreeBSD 4.11 today? Well, most people would probably respect historical interest and in fact I was keen on tinkering with a legendary old 4.x release and see if you can have some fun with it (spoiler: Oh yes, you can!). But this would be a task for times when I have absolutely nothing else to do. So it’s fairly obvious that there’s another reason.

“Old habits die hard” they say and there’s definitely truth to it. Another truth, however, is that in the IT world things often don’t go away easily. There’s still a considerable number of machines out there that run DOS. In production. Doing important stuff. Sometimes just because it works sufficiently well. But often they have special tasks that for some reason or another cannot be migrated as nobody has any idea clue how you’d do this.

You want examples? Sure. The first example is “Game of Thrones” author George R.R. Martin writing his books using WordStar 4.0 on a DOS machine. A lot of companies use DOS, too, but you usually don’t hear about that. And less than a month ago a new version of FreeDOS (v. 1.2) was released.

Or do you remember the French airport that was having trouble due to using Windows 3.11? That was in 2015 and they announced that they were planning to migrate their system officially by 2017 (but expectedly by 2019 or even 2021)!

FreeBSD 4.11

One such case of a formerly very popular operating system that just won’t go away is FreeBSD 4.11. From the version number alone you can see that this is a very special one: Usually FreeBSD has three to four point releases in a release’s life cycle. FreeBSD 4 obviously was different. The main reason for that was that version 5 kind of was to FreeBSD what version 4 was for MS-DOS: Innovative in its day but disappointing especially in terms of stability (and performance) among other things.

I have to take up the cudgels for FreeBSD 5, though. It blazed a trail for important technology that we rely on today like the GEOM framework, porting the system to amd64, and so on. Implementing things like that meant digging deep into the system and doing pretty invasive changes. It totally paid off in the long run but back in the day people tended to avoid the early 5.x releases or the whole release cycle altogether.

There are many reasons why some companies still have FreeBSD 4.11 running: Critical systems that cannot be migrated, installed programs to which the source code was lost, etc. Things like that. Most of these systems are probably used internally only but I’m pretty sure that there’s also quite some 4.11 machines still serving data on the net…

Now what others do (and what their reasons for it might be) are not my primary concern – I can’t do anything about it anyway. However the company that I work for has its share of legacy systems, too. Especially younger admins hate them. Actually they are impressively stable and rarely if ever have problems. In fact they keep running, silently doing their job. But now and then you have to do some changes to them and that’s where the trouble starts: They simply cannot be compared to the modern-day Linux systems that we’re used to administer daily!

Preparations

I kept wondering if that situation could be improved. It wouldn’t be worth the effort to do this during work time but I was curious enough about the legendary 4.11 release to give it a try. There’d surely be enough to learn. And should I succeed in bringing a test system into a more modern state there would also be a benefit for us at work.

We still have all those old CDs at work but I made my decision impulsively on Saturday. Fortunately ISOs of the old Walnut Creek CD-ROMs are available for free download. Thanks, Archive.org! I burned it to a CD and was able to boot from it.

I actually installed the OS on real hardware. First on an old HP compaq nx9010 laptop that I got for free a while ago. Things worked pretty well, but since what I’m trying to do is almost all about compiling software, the compilation times proved to be a bit long. Therefore I decided to redo the previous steps on faster hardware – and since amd64 PCs are capable of running i386 programs, all was well. I just had to disable AHCI in the BIOS since FreeBSD 4.11 doesn’t know what to do with that.

The actual installation was interesting enough (for people like me who enjoy history) and therefore I decided to redo it again in VirtualBox to take screenshots and describe the installation process for the remainder of this post.

Installation

Quite some time has passed since FreeBSD 4.11 was released in 2005. More than a decade is a lot in a man’s life but it’s a hell of a lot when it comes to software and operating system development! I had never before installed any version of FreeBSD older than 7.4 and even thinking about the differences between 7.x and 11, many things changed. But what would 4.11 look like? Would I be able to install it without difficulties (I tried a very old version of OpenBSD once and quickly gave up because I didn’t really like the idea of manual partitioning of the drives…)?

FreeBSD 4.11’s Kernel Configuration Menu

I was surprised to run into something that I had never seen even before the installer started! There’s this Kernel Configuration Menu where you can choose to either boot the standard kernel or to customize it. Of course I was curious and so I decided to take a look at it.

FreeBSD 4.11 kernel drivers selection

This brings up a friendly little tool that allows for easy navigation through the driver tree. The screen is basically split into two parts: Active drivers and inactive drivers. Each one consists of several categories which are collapsed and can be expanded for a list of drivers.

Browsing FreeBSD 4.11’s additional network drivers

The standard kernel worked on both of my machines and so I didn’t play around with this after exploring it briefly. When the kernel had booted, I found myself in the good old Sysinstall utility. Yes, it’s pretty ugly and it looks a bit complicated – which is due to the fact that despite it’s name it does a lot more than just handle the installation. Yet in fact it’s really easy to navigate through.

I like the Express option that lets you perform an installation as quickly as possible by asking you only the absolute minimum of necessary questions. This goes as far as not asking for a root password and leaving it blank on your first login! Fair enough, it’s easy to set one yourself. Current versions of FreeBSD do not have this option. That may be due to the fact that it’s not terribly helpful for setting up a serious system. And the use case that I have for it here is definitely more of an exception!

FreeBSD 4.11’s Sysinstall: Main Menu

The first thing that’s essential is of course disk partitioning. If you don’t require any special setup, you can simply use the whole disk. I was glad to see that this can be done by simply pressing “A”!

Installing FreeBSD 4.11: Disk partitioning

Just like today FreeBSD 4.11 allows you to choose if you want to install a boot manager, just write the boot loader or don’t do anything. The first option can be used for dual-boot systems where you want to load either FreeBSD or another OS. The last option won’t work by itself; if you choose that, you need to work with an external boot manager that supports FreeBSD. In my case just installing the standard loader makes most sense.

Installing FreeBSD 4.11: Choosing boot options

Then there’s creating the disklabels. This is another thing where I’m happy to find an auto defaults option (simply pressing “A” again) because I just wanted a quick test system anyways and didn’t want to think too much about the disklabel layout!

Installing FreeBSD 4.11: Editing disk labels

Finally we have to select what to install. To make things easy, sysinstall offers several distribution sets to choose from. Minimal is all that I need for my experiments, but back in the day a lot of the other options were definitely quite handy. Two things stand out: You can choose to install FreeBSD + X11 for example. Current versions of FreeBSD don’t let you do that. You’ll always get a console only system and have to install X11 yourself (which IMO is something that leaves the barrier unnecessarily high for new users who don’t know FreeBSD nor its packaging system, yet!).

Sysinstall also allows for custom distribution selection. This gives you access menu that allows for really fine-grained choices. Again, FreeBSD’s installer cannot do that today. While it’s probably nice to have that option, it’s not a must-have. People who have use cases for that will probably know what they do and install FreeBSD manually, anyway. Still I like having it in the installer!

Installing FreeBSD 4.11: Choosing distribution sets

The only thing left is selecting what medium to install from – the CD in my case. Formatting a large drive takes quite a moment because we’re talking UFS1 here! Don’t start looking for a way to use UFS2, it’s not available in FreeBSD 4. The actual “minimal” installation is quite quick and just a moment later I can reboot and the system comes up just like you’d expect it to. Now that was the easy part of my adventure. Next stop: Brushing the dust away and updating that beast(ie)!

3+ months on TrueOS – a critical write-up

My journey into the *nix world has not been a completely straight forward one. I’ve used Linux (various distributions) exclusively for quite some time before I felt that more and more things were heading in the wrong direction. Sure, it’s all open source and I could do things my own way. In fact I did roll my own distribution for a short period of time but this was more because I wanted to experiment with things. And while there are examples (like Void Linux) that prove that a single person can keep a serious distro running, I’m far from having the knowledge nor the time (or even the urge) to do so. But even if I had all that there’s something wrong with parts of the Linux ecosystem and community that I don’t feel is fixable at the moment.

For those reasons I was pretty open to new things when I encountered actual FreeBSD servers at work. I came to love the *BSD way of doing things and used FreeBSD and OpenBSD systems on laptops and in VMs to play around with. In January I decided to put PC-BSD on my main machine but had to leave it for Linux again pretty soon, for various reasons. Over the time I really wanted my BSD back and get rid of the Linux trouble (that used not to bug me as I knew nothing better). I’ve written about my experience with PC-BSD and Linux again in my previous post in some detail. The time to give a FreeBSD based desktop another try came when TrueOS was released. So that’s what this article is about: Using TrueOS as your daily driver for some months!

TrueOS in general

When I found out about TrueOS sometime in July, I was very curious how different it would be from PC-BSD. So I downloaded the ISO and installed the OS on my primary machine (no risk no fun, right?). The installation went as smooth as it did with PC-BSD. My hardware was supported. It seemed like a good start and I thought that I’d probably only need a few weeks to adapt to Lumina and be a happy TrueOS user. That was a bit too optimistic. My experience with this system really is a mixed bag. It’s not all bad nor is it all great. There’s light and shadow. And to be honest: While it’s pretty much ok as a daily driver (it’s mostly stable and most features are there), I think it feels quite “beta” currently.

TrueOS installation: Simple but shiny

I tried to post problems that I had with it on the PC-BSD forums but didn’t have too much luck with that. One post has not had a single answer (after several month now). Another was hijacked by somebody else and then closed by a moderator even though my issues (being the thread starter!) had not been resolved. While I originally wanted to open bug reports after getting into the community on the forums, that experience didn’t really help holding up my motivation. Yes, I know: I should just have reported things in the proper place. But I’m a person who wants to have a deeper relationship to his OS. I don’t just want to use it. I want to be part of it! For that reason it’s important to me to get into the community around an OS early on. For me this did not work out on TrueOS. I didn’t feel like I was getting anywhere on the forums and I didn’t succeed in feeling at home on the OS level, either.

But that’s personal stuff. Let’s start with going into a bit of detail on some areas of the system, shall we?

Bad: Graphics acceleration

Most of the bad points that I mentioned for PC-BSD (see the previous post) were still valid: Startup time was a bit long, the system is rather heavy on the battery, I didn’t find a way to log in multiple users graphically, etc. But there’s more. I think that the single most important problem that TrueOS suffers from is the fact that graphics acceleration had been broken for quite some time now. It worked when I first installed the system. Up to the update on 9/11 things were fine, too, but the next one broke it and while I hoped that it’d be fixed soon, I’ve waited for that ever since.

A freshly installed Lumina desktop

So if I wanted to watch a video I had to reboot and select the boot environment from 9/11. This also affected my work with VirtualBox. Probably thanks to that issue I could no longer switch my VMs to full-screen mode which was rather annoying. So I often had to reboot and use an old BE to be able to actually use my PC as intended. The fact that after a Lumina update the old BE would no longer be able to show the panel didn’t help either… 😦 I love BSD and want it to succeed. Only for that reason I’ve been tolerating things like that (which mean major inconvenience, honestly) for quite some time. But certainly you don’t attract many new BSD users with a system that has such issues! I really, really hope that this one gets fixed soon…

Good AND bad: Display settings

Lumina comes with a monitor configuration utility which is an excellent program – at least in theory. What makes it great is that it not only allows you to set the resolution. It also offers a simple and elegant GUI solution to manage multiple monitors! Great stuff, I love that. My primary PC is a laptop that I mostly use at home with an external monitor instead of the laptop screen. The monitor configuration allows me to add the external monitor and deactivate the other one. I’ve played with this a bit and it all seems very nice and mature.

Adding a second screen in the ‘Lumina Monitor Configuration’

However… It simply won’t keep the settings! That means was always 8 clicks after each start before my monitor setup was the way that I needed it and I could begin to do some work. I have no idea why it won’t save the settings and remember them next time. This is completely unnecessary.

Screen resolution configuration

I’ve also noticed a little consistency issue is that most of the Lumina desktop is localized properly (for the screenshots here I’ve used an English system, though!). The monitor control for example isn’t. I speak English and thus I have no problems using it but it makes the tool feel a bit out of place.

Neutral: Insight

I’ve tried to become friends with Lumina’s file manager, Insight. It didn’t feel right at the beginning and sometimes it decided to dump core when I accessed a new directory. But I finally kind of got used to it after some time.

The old Insight file manager with the side bar

Then another update happened… Yeah, the new git functionality may be nice. But guys, really… What did you do to that sidebar with the icons for copy, cut, paste, etc? The one thing that made me accept Insight was gone all of the sudden and I have been unable to get it back… I must admit that it’s probably not that bad of a loss as shortcuts were finally implemented. Still it didn’t take that much space and for people who don’t use shortcuts (I’ve seen a lot of such people!) it still was a very nice feature. No idea why it was dropped entirely instead of making it opt-in. Same thing about the “action buttons” and the “show thumbnails” option. It’s simply gone.

The new Insight with dual column mode

Another update added the dual column mode. That’s pretty useful IMO. But it happened after I stopped using TrueOS as my main operating system.

Good: SysAdm

This is one aspect where TrueOS really shines. Initially it felt quite empty and unpopulated but same of the updates added more and more options to it. SysAdm is a middleware that exposes an API to manage FreeBSD based machines locally or remotely.

The ‘Sysadm’ client for the local machine

I’ll be keeping an eye on this and look forward to install it on a FreeBSD server and try out remote management! But even the local client comes with a very nice GUI that has a lot of functionality now. Thanks and great work on that one TrueOS team!

Bad: Window manager

TrueOS uses the Fluxbox window manager with Lumina. Some people like it, some people don’t. I’m on the side of those who don’t but that’s not my main problem. People who’ve used *nix systems for any period of time probably know more than one wm and simply switch over to one that they like. The trouble is: It looks like Fluxbox is not meant to be replaced when you run Lumina! There’s no easy way to configure a different wm and in fact I didn’t find anything at all.

Lumina desktop settings

Worse: There are standards (the ICCCM in case of window managers). Following standards makes sense. Fluxbox doesn’t follow them. Window managers are meant to let other wms “take over” if you run your-favorite-vm –replace on the console. Fluxbox won’t cooperate which is very unfortunate. To replace it with sawfish (my wm of choice) I need to kill fluxbox first and then start the other wm… That’s not cool.

However I can fully understand that the small team that brings us TrueOS concentrates on supporting only one wm. Using sawfish I’ve experienced repeatable crashes (especially with Insight) where the system proved to be stable otherwise. And there’s another reason not to take this point too seriously: Fluxbox is not here to stay. Ken Moore has stated that he’ll write his own window manager to work perfectly with Lumina. So at some point Fluxbox will be replaced. I’m looking forward to this and hope that it’ll be a better replacement.

Neutral to good: Lumina DE

One of the core features of TrueOS is its native Lumina desktop. It was written from scratch, is BSD licensed and one of its design goals is being light-weight. Sounds excellent, doesn’t it? You bet it does! But does it live up to the high expectations? Like the whole TrueOS project its a bit of a mixed bag… First: A permissively licensed “BSD first” desktop is a dream come true. And I’m all for it being light-weight! The only problem here is… it isn’t.

It probably depends on what you compare it to. Sure thing: Compare it to KDE and you will find Lumina pretty light-weight. Then again – good luck finding any *nix DE that’s more heavy-weight than KDE is! If you compare Lumina to other desktops that state that they’re light-weight (be it Xfce, LXDE or even EDE), it clearly is quite a bit on the heavy side. In fact that’s no surprise due to the choice of toolkit that was made. Qt is the fattest toolkit out there. It does have it’s good parts, but being light-weight is nowhere near its strong points. However KDE (which uses Qt, too) has been the default DE of PC-BSD before and so Qt is what the TrueOS team knows best and that makes this toolkit a sensible choice despite the downsides.

Lumina panel configuration: Nice and flexible!

What Lumina does very well is being flexible. You can configure the menu the panels and just about everything to your liking. And you can do so using the GUI instead of having to edit config files. Even better: It’s pretty easy to do and after playing around with it a little you soon know how things work and where they are configured. Two thumbs up for that! I just miss the ability to right-click the panel to configure it. That’s probably the first thing newcomers try as everybody expects it to work.

Lumina is a desktop environment with lots of potential that already works quite well. It isn’t my favorite desktop and it does have some issues right now. However it works well and there are definitely people who prefer it over any other DE. And I’m pretty confident that it will continue to improve.

Good: Bootloader with BEs

PC-BSD used a modified version of GRUB2. While that program certainly works it’s not exactly my cup of tea. As stated above, I like light-weight software. And a bootloader that bears Grand in its name (and rightfully so) is not really my first choice. It’s alright if you need to dual-boot Linux or something but for just booting FreeBSD its more than I want.

TrueOS’s Bootloader – it supports BEs!

For TrueOS the team has decided to migrate back to the default FreeBSD bootloader after years of using GRUB. Excellent choice! Especially since it can now also use Boot Environments.

Good: PCDM

When it comes to display managers, I’ve come to like LXDM on Linux. Unfortunately it uses some linuxism in its code and for that reason could not be ported to *BSD. TrueOS offers another gem in this regard: PCDM. It’s a program that let’s you log in conveniently, providing all the features that you expect from a decent display manager – and more.

TrueOS’s display manager: PCDM

In PC-BSD I remember that I was initially unable to log in due to using a character that’s on the German keyboard but not on the US one. With TrueOS I no longer experienced such problems. On the contrary: I learned an alternative keyboard layout a while ago that offers better ergonomics and lets me type lots of foreign characters, too (e.g. the whole Greek alphabet). With PCDM I can use it to log in (allowing for nice, strong passwords, yay!).

Changing the keyboard layout in PCDM

I don’t have a 4k monitor, but it’s nice to see that PCDM is prepared for 4k already. The display manager lets you select various DPI options so that you don’t have the feeling of sluggish mouse movement when you use high resolutions.

PCDM’s DPI options: Ready for 4k

The only thing that bugged me quite a bit: The display manager was only displayed on the first screen, forcing me to open my laptop again to log in. I suggest to just show the login manager on every screen; this would be much more convenient.

Neutral to good: Update

The update mechanism of TrueOS has some advantages over the common desktop update methods. If you begin an update, it only fetches all packages and lets you continue to work in your current session. When you are about to shut down the system, it then asks if it should install the updates. Accept that and the system will shutdown the desktop and graphical mode and start updating. While it is doing that it’ll tell you not to turn off the computer and to change to the second TTY to see the details of the update process. When it’s done, the machine is powered down (or restarted if you chose that) like normal.

System update on TrueOS

But the really special thing is how the update is performed. A new Boot Environment is created, all packages are deinstalled and then reinstalled in their newest version. This has two advantages: It is the cleanest possible approach and it means that you can go back to any previous state by just selecting the respective BE! If you plan to do that, be sure to configure how many BEs the TrueOS update system keeps. Otherwise it will trash old ones (which may be a sensible default for space reasons).

What’s the downside? TrueOS is not a tiny operating system. Downloading all the packages with every update (can anybody say Noto fonts?) will require quite a few bits going over the wire. If you’re in a place with a slow connection, or worse, you have a monthly limit of how much you can download, then TrueOS’s way of doing things might not be for you.

Good: OpenRC

Ah, the bliss of a new init system! I’ve waited quite a while for that to happen. And when does it happen? About a week after I replaced TrueOS on my computer with another BSD operating system, Kris Moore (founder of PC-BSD/TrueOS) announces that they’re in the middle of switching over to OpenRC!

Truth be told: I didn’t expect this. When I heard Kris talk about nosh on the BSDNow show, I suspected that they might build that into PC-BSD. Then I had the impression that the team favored relaunchd (now renamed to jobd). And now it looks like OpenRC has landed!

TrueOS starting using OpenRC!

I’m fine with that as I already know it from Gentoo Linux. While I think it would have been interesting to see one of the other options get some attention, it now looks like OpenRC becomes the most significant alternative for people who don’t want to use Systemd! Maybe that’s not a bad thing at all.

While I don’t think that FreeBSD is going to adopt this change in the forseeable future (the BSD init system works well for servers after all) it totally makes sense to speed up the starting process for desktop machines. Thank you, TrueOS team! This takes care of another issue where FreeBSD just couldn’t compete with Linux.

Major differences from PC-BSD

The applications that come with TrueOS are pretty standard now. Initially the team provided some Qt5 alternatives like a browser I had never heard of and such. It’s a good decision to provide applications that people know but I also like the spirit of trying out new things and see if they work out. TrueOS in general is even more open to experiments than PC-BSD was – and even that was quite the opposite of a boring OS!

Obviously TrueOS has a new name. There are people who don’t like it. Some claim that it can be mistaken for another OS. But let’s be honest: Does “the average user” even know that there’s Tru64 UNIX? Most likely not. And people who know that it exists probably have enough *nix knowledge to tell the two apart. Other people criticise that it’s a bit of a big-mouthed name. Maybe it is but I don’t really care so much about that. In times of PC-BSD the server variant was already called TrueOS and for their new system the team wanted to rebrand the project without using a completely unknown name and so TrueOS actually was a sensible choice.

The biggest difference between the two is that PC-BSD was built from FreeBSD releases originally and ultimately headed towards the -STABLE branch. TrueOS takes this one step further and builds upon -CURRENT! This means that you always get the latest drivers and newest stuff but you may have to live with problems like broken graphics acceleration…

Another difference is that while PC-BSD begun its life supporting KDE and later leaving the user a choice between several desktop environments, TrueOS concentrates on the Lumina desktop. Some other DEs can be installed if you wish, but Lumina definitely is the standard.

And then there’s the newest addition to the project is TrueOS Pico, a variant meant to build ARM based thin-clients as well as a “Pico Server”. A very intriguing concept!

Conclusion

Looking back on more than three months of daily TrueOS usage, I must say that I went through highs and lows. There’s the painful moments where I had to grind my teeth and force myself to carry on. And then the opposite happened and I come across something that’s just amazing. All of that makes it not too easy to draw a clear conclusion. TrueOS is evolving at a very fast pace and at the present time my conclusion is that it is a unique OS that might work for you. There are operating systems where that’s more likely than with TrueOS but I’d bet that they are also far less innovative.

The TrueOS project is relatively young. I wouldn’t bet on even the team leader to know exactly where it’s heading. This is a good thing for us all, even for people who do not plan to use TrueOS. Why? Because it is not afraid to try new things and by doing so will continue to push FreeBSD forward in the desktop field!

PC-BSD’s goal was to provide a FreeBSD powered desktop that’s both easy and convenient to use for seasoned and new users alike. The rolling release character of TrueOS may not fit the former audience completely. It will be very interesting to see where the project will eventually find its place. Long time FreeBSD users who want the newest features on their desktop? BSD enthusiasts who want enjoy a permissively licensed desktop OS? Who knows! Time will tell (those who keep an eye on it). I may have switched to another OS for my daily work but TrueOS is far too exciting a project to just abandon completely. And if you love BSD you may want to give it a shot now and then. If you haven’t already tried it, correct your ignorance and download an ISO now!

Back and forth: Linux and *BSD

This is kind of the post that I wanted to write much earlier this year. After running a Linux-only environment at home for years, I had become less and less happy with the general direction things seem to be heading. I had run FreeBSD and OpenBSD on real hardware (old laptops) and several versions of PC-BSD in VirtualBox over the years. In January I decided to step forward and install PC-BSD (10.2) on my primary computer for daily usage. It remained a short episode – and this post will describe why. When TrueOS was released to the public I decided to try out that right away. But that will be another post.

Initial contact

I cannot remember when I first read about the BSDs. That must have been many years ago when I became interested in reading a bit about UNIX. I remember beastie and puffy and I remember that I failed installing a system in a VM because it was somehow too complicated. It likely was OpenBSD and the chance is quite high that I quit during the partitioning which probably was way over my head at that time.

While I never lost interest in it (Unix fascinated me) I decided to “learn Linux first” as that was the system I had chosen to run my computers with. As the Linux world was big enough for years (trying out the various desktops, doing a lot of distro hopping, …) I touched *BSD only rarely. Basically it was limited to installing PC-BSD in a VM when I found out that a new version was released. It seemed to be nice but I didn’t see any benefit over my Linux systems and so I stuck with that.

After studying something entirely different, I had made the decision to break up and get into the IT instead, even though was I well beyond the age that you usually start an apprenticeship. In my country that means that you apply to a company to work as an apprentice there half of the week and go to school the other days. Being somewhat of a Linux nerd I had only applied to companies that I knew weren’t using Windows – I had left that mess and was determined to avoid it in the future as far as possible. In the end I signed a contract of apprenticeship with a hosting company, moved into the area and started learning Linux a lot deeper than I had before. And… I came in contact with FreeBSD.

Being a hosting company that had been founded in the nineties, it had of course started on FreeBSD. Even though the focus of the company shifted to Linux years ago, there still were about 100 servers running FreeBSD. My colleagues generally disliked those servers – simply because they were different. And our CIO declared that he hated them and would love to get rid of them as FreeBSD was totally obsolete these days. If it hadn’t been for our boss to have a soft spot for them (as that had been what he started with and also what he had come to know best over the years) there definitely would have been far less FreeBSD servers around.

Digging into FreeBSD

Now for whatever reason I do have a heart for underdogs and so I begun to be interested in those odd systems quite a bit. Nobody wanted to touch those dinosaurs if he didn’t really have to. However somebody had to take care of them anyways, right? They were production servers after all! I volunteered. There were moments where I kind of regretted this decision but now in hindsight it was an excellent choice. I’ve learned a ton of little things that made me understand *nix and even the IT in general quite a bit better compared to what I would know now if I had followed the straight Linux path.

I also found out that only very few things that the colleagues hated about our FreeBSD boxes were things to actually blame FreeBSD for. By far the biggest problem was that they simply had been neglected for like a decade? Our Linux systems used configuration management, the FreeBSDs were still managed by hand (!). We had some sophisticated tooling on Linux, on the BSD boxes there were crude old scripts to (kind of) do the same job. Those systems were not consistent at all; some at least had sudo others made you use su if you needed to use privileged commands… Things like that. A lot of things like that. So it wasn’t exactly a miracle that the BSDs were not held in very high regard.

As I said, I didn’t really see any real advantage of BSD before. Linux even seemed to be easier! Think network interfaces for example: “eth + number” is easier than “abbreviation of interface driver + number”. But Linux has since moved to “enp0s3” and the like… And when you think again, it does make a lot of sense to see what driver an interface uses from the name. Anyways: I begun to like that OS! FreeBSD’s ports framework was really great and I realized the beauty of rc.config (Arch Linux did away with their central config file to get systemd. What a great exchange… – not!). Also I liked the idea of a base system quite a bit and /rescue was just genius. Would my colleagues lose their contempt for our BSD servers if they were configured properly? I thought (and still think) so.

My apprenticeship was nearing its end and I had to choose a topic for the final project work. I was advised to NOT do something Linux related because the examiners… *cough* lacked experience in that field (in the past an apprentice even failed because they have no idea what they are doing. He went before court and it was decided in his favor. A re-examination by people who knew Linux got him an A!). Now things like that make me angry and calls upon the rebel in me. I handed in a FreeBSD topic (evaluating Puppet, Chef, SaltStack and Ansible for orchestration and configuration management of a medium-sized FreeBSD server landscape).

So for servers I was already sold. But could *BSD compete on the desktop, too? I built two test systems and was rather happy with them. However I wanted to try out a BSD system optimized for desktop usage. Enter PC-BSD.

Working with PC-BSD

I was called nuts for making that switch just days before the final presentation of the written project work (“you need to pass this – your entire career depends on it!!”). But I didn’t want to do a presentation on a FreeBSD topic using a Linux machine! Well, in fact I had been too optimistic as the installation turned out to be… rather problematic due to a lot of bad surprises. To be fair: Most of them weren’t PC-BSD’s fault at all. The BIOS mode on my computer is broken in it not supporting booting off GPT partitions in non-UEFI mode. This lead to my drives disappearing after installation – and myself wondering if my classmates were right… Never change a running system! Especially not if you’re pressed for time!

After I found out what the problem was, installing to MBR was an easy thing to do. I still needed every single night that I had left but I got everything to work to at least the level that allowed me to hold my presentation. Another thing was that I had enabled deduplication on my ZFS pool. “24 gigs of memory should be enough to use that feature!”, I thought. Nobody had told me that it slows down file deletion so much that deleting about 2 GB of data meant to go and do something else while ZFS was doing its thing. Even worse: The system was virtually unresponsive while doing that so you could forget browsing the web or something like that in the meantime. But truth be told this was my own mistake due to my very own ignorance about ZFS and I can hardly blame PC-BSD for it.

I kept PC-BSD on my laptop for about 1.5 month before I needed to return to Linux – and I would in fact even have returned earlier had I had the time to reinstall. While some issues with PC-BSD vexed me, too, I could have lived with most of them. But my wife complained all the time and that of course meant the end for my PC-BSD journey.

So what were (some) of the issues with it? My wife mostly uses the PC to check email when our children are occupied with something for a moment. For her the very long boot time was extremely annoying. And really it took multiple times as long as the Linux system before (and that was still one with Upstart!). Keeping one user logged in and changing to another user quickly wasn’t possible – which meant that I had to shut down my multiple virtual machines and log out completely if my wife just wanted to quickly check mail or something. Not cool. Things like that.

And then there were a few things that annoyed me. It drew power from the battery much, much faster than the previous Linux system. When watching a video, the screen saver kept interrupting it. Firefox had strange issues from time to time and liked to crash. Working with EXT4 formatted disks was a pain. And so on and so forth.

Of course there were good parts, too. I had a real FreeBSD system at my hands with access to ports. Two firewalls (that are nothing like the mess that is netfilter/iptables!) to choose from. Excellent documentation. Nice helper tools (like the automounter, wifi manager, disk manager, etc.). Several supported desktops to choose from. And of course the well thought-out update system that I liked a lot. Thinking about it, there are a lot of good parts actually. Unfortunately even a ton of things nice to have have a hard time covering things conceived as no-gos. That’s life.

I had intended to update to 10.3 and then write a complete blog post about PC-BSD. My wife didn’t like the idea much, though. In addition to that I had little spare time and no alternative spare hardware, so there wasn’t a chance for me to actually do that.

Interlude: Linux

So it was back to Linux. With systemd this time. I’m not exactly friends with that omnivoristic set of tools that annoyed me perhaps just not enough to switch the system over to runnit or openrc. Other than that life was good again (as my wife was happy and I could do my work). But there was one thing in the short period of time with PC-BSD that had changed everything: I had caught the bug with ZFS!

Fourtunately there’s ZFSonLinux, right? So I installed that and created a pool to use for my data. In general that worked but it’s a bit more hassle to set up compared to FreeBSD where you basically get it for free without having to do anything special! If you don’t want to compile all packages related to ZFS yourself for each new kernel, there’s a third-party package repository for Arch. ZFS is not in the official ones. At some point the names of the packages changed and the update failed. I didn’t find anything about that and had to figure out myself what happened.

After another kernel and ZFS update that I did in the morning succeed. But when I came home, my wife told me that when she logged in, she was logged out again almost instantly. I booted the computer and logged in – the same thing happened. What was that? No error message, no nothing. The system simply dropped me back at the login manager… So I switched to text mode to take a look at what might be wrong with the system. Long story short: My pool “homepool” which held all user’s home directories was not available! And worse: zpool import said that there were no pools available for import… With the update, ZFS had stopped working! That hit me in the wrong moment whan I had very little time and so I had to downgrade as the quickest solution.

In the end I chose to compile the “solaris porting layer” and the other packages myself. This was not so bad actually but knowing that on FreeBSD I’d have access to ZFS provided by the operating system without having to do anything (and that nobody was going to break it without it probably being fixed again in no time) vexed me. Of course there were other things, too, and using FreeBSD on other boxes, I wanted it back on my main desktop machine as well.

What’s next?

I installed TrueOS and used it for over three months. The next post will be a critical writeup about TrueOS.

Bacula on FreeBSD (pt. 5): A day at the pool

Edit: I put up the original post only for a short time before I decided that I wasn’t really happy with it at all. So I took it down again to rework it a bit. By now I’ve more or less rewritten it completely. I hope that this version will be more useful.

This is part five of my Bacula tutorial. The first part covered some basics as well as installing Bacula and starting all three daemons. Part two dealt with allowing all components of Bacula to interact with each other locally, debugging config problems and doing a first test backup was.

In part three the configuration was cleaned up and split into smaller parts, the first self-created resources (fileset, device and storage) were added and a backup job customized using the bconsole. Part four detailed restoring files from backup and discussed jobs as well as volumes, labels and pools.

Part five will show how to create a new storage pool. We’ll use memory disks so that we can simulate partitions. Then we’ll talk about jobs again and have a closer look at volumes in our pool and some of the states they can have. Finally we’ll briefly touch upon the topic of recycling volumes. You’ll need to know all that (and actually some more) prior to planning your real pool(s).

The fifth part continues where the third one left off. During this tutorial series we use VirtualBox VMs managed by Vagrant to make things as simple as possible. If you don’t know how to use them, have a look here and here for a detailed introduction into using Vagrant and on how to prepare a VM template with FreeBSD that is used in the Bacula tutorial.

Preparations for a storage pool

Load the latest snapshot, enter the VM and switch to root:

% cd ~/vagrant/backuphost
% vagrant snapshot restore tut_4_end
% vagrant ssh
% sudo su -

Our VM has a simple partition scheme where essentially everything is put in one partition. A volume that provides storage backed by a file will keep growing until the disk is full. To enforce restrictions we have to create a pool. Also our VM certainly does not have a tape drive or anything like that. Still I’d like to simulate filling a medium! How can we do that? Probably by creating virtual disk-backed storage for which we can set a fixed size.

This time we’ll backup a directory with a bit more data in it. I suggest /usr/bin. Let’s see how big it is (mind this size! Especially if you’re using a version newer than the 11.0 release that I’m using here the size may be different):

# du -sh /usr/bin

143M /usr/bin

Ok, we’re going to create a sparse file of 300 MB in size next (so that at a bit more than two full backups would fit in there) as well as a directory for the mountpoint:

# truncate -s 300M /var/backup/ufs0
# mkdir /mnt/storage0

Now we need to make the storage accessible as a device rather than just a file. FreeBSD calls these devices memory disks because most of the time you’ll want such a device to be backed by RAM (to benefit from its speed), but file-backed storage is also possible. As a first step we need to create the device and, as a second one, put a filesystem on it. Then in step three we can mount it on a directory to make it available on the machine’s global filesystem hierarchy. Fortunately FreeBSD is pretty good in doing memory disks and comes with a command that can do all three steps at once:

# mdmfs -F /var/backup/ufs0 -S -n -p 700 -w bacula:bacula md0 /mnt/storage0

Let’s check if that worked:

# mount

See the mount? Looks like we have our file backed storage in place.

Configuration changes

Now we can create a new fileset to back up /usr/bin:

# cd /usr/local/etc/bacula
# vi includes/dir_fileset.conf

Add the following lines at the end of the file:

FileSet {
Name = "usr-bin"
Include {
Options {
signature = MD5
}
File = /usr/bin
}
}

Save the file and exit. Of course we need to tell the sd where to store the backups – we need to create another device:

# vi includes/sd_device.conf

Add this device resource at the end of the file:

Device {
Name = Stor0-Test
Media Type = Test
Archive Device = /mnt/storage0
LabelMedia = yes;
Random Access = Yes;
AutomaticMount = yes;
RemovableMedia = no;
AlwaysOpen = no;
Maximum Concurrent Jobs = 5
}

And the director needs to know about this storage as well:

# vi includes/dir_storage.conf

Add this block at the end of the file:

Storage {
Name = Test
Address = localhost
SDPort = 9103
Password = "sdPASSWORD"
Device = Stor0-Test
Media Type = Test
Maximum Concurrent Jobs = 10
}

We’ve done all this before but the next step is new. We’re going to create a pool:

# vi includes/dir_pool.conf

Again add the following lines to the file:

Pool {
Name = Testpool
Pool Type = Backup
Recycle = no
Maximum Volume Bytes = 90M
}

Just set recycle to “no” for now – you’ll see what that does in a minute. We also want to force a maximum volume size of less than what the backup is worth of data so that two volumes will be needed for one full backup. And to avoid having to use “mod” at the bconsole all the time, we’ll add a new job for convenience:

# vi includes/dir_job.conf

Here’s the job resource to put at the end of the file:

Job {
Name = "TestJob"
JobDefs = "DefaultJob"
Level = Full
FileSet = usr-bin
Storage = Test
Pool = Testpool
}

Jobs

Ok. Our basic preparations are done. Let’s restart the daemons and then try and see what happens if we run our new backup job:

# service bacula-dir restart
# service bacula-sd restart
# bconsole
* run
4
yes
* mes

[…]
29-Oct 12:48 backuphost.local-sd JobId 4: Job Testjob.2016-10-31_12.48.33_03 is waiting. Cannot find any appendable volumes.
Please use the “label” command to create a new Volume for:
[…]

The job cannot start because there are no volumes that Bacula could use. We could now create volumes using the label command as we’ve done before. But we configured our device with the LabelMedia directive set to “yes”! It should create volumes automatically as needed! Time to look at the configuration again. But let’s first take the chance to cancel the currently pending job.

We already know the JobId from the message above. But let’s pretend we didn’t know. We’ll ask the director for an overview of jobs (both past and present):

* list jobs

There we have a simple table. Let’s see what information it holds. First we have a unique JobId for each job. The second column holds the name of the backup job – this will usually be the name of the client or RestoreFiles for a restore job. But as you can see in our case, “TestJob” will work as well (however you really should stick to names that hint which client they belong to in a production environment as things would get pretty confusing really fast). StartTime is self-explanatory. Type is “B” for backup and “R” for restore in our example. We’ve only run full backup level jobs so far. JobFiles and JobBytes is self-explanatory again. And in case of JobStatus, a “T” means terminated, an “R” running, “A” for aborted, etc. Now we’ll cancel the job that is waiting for a new volume:

* cancel 4
yes
* mes

[…]
Backup Canceled
[…]

So we’ve canceled the job. Let’s exit the console now and take a look at the pool configuration again:

* exit
# vi includes/dir_pool.conf

Our pool resource is missing a directive that specifies how the label names are to be composed. We should add that one line real quick:

Label Format = "Test-"

Now we need to restart the dir and invoke the bconsole again.

# service bacula-dir restart
# bconsole

Turn up the volume(s)!

Then we can take a look at the volumes that we have so far:

* list volumes

[…]
Pool: Testpool
No results to list.
[…]

The pool Testpool is empty right now. Take a look at the pools used for our previous jobs and get an idea of what they look like. Now let’s run the test job again and see what happens:

* run
4
yes
* mes

[…]
Labeled new Volume “Test-0003” on file device “Stor0-Test” (/mnt/storage0).
[…]

So auto labeling obviously works. This is one huge benefit of using a pool! But there are others like limiting the volume sizes and many more. Did the backup complete successfully by now?

* mes

[…]
Backup OK
[…]

It did. Time to look at the volumes again; look for the column VolStatus:

* list volumes

[…]
Full
Append
[…]

The first volume has a VolStatus of Full and the second one is Append which means that more backup data can be written to it. We’ll do that by simply running our test job again:

* run
4
yes
* mes

[…]
WARNING: device is full! Please add more disk space then …

Please mount append Volume “Test-0005” or label a new one for:
[…]

We’ve created a really small ramdisk for /mnt/storage0 and it is full before the second full backup could be completed. But how’s that? There’s 300 MB of space and the fileset backs up less that 150 MB! Is there so much overhead? No, not really. What has happened here is that we restricted the pool to volumes of 90 MB each. The three volumes occupy 270 MB – and while there’s 30 MB more left on the pool, that’s too little space to create another volume on it! So what do we do now? First have a look at the volume list again:

* list volumes

[…]
Full
Full
Full
[…]

Coming full cycle

All of them are listed as Full. But there’s some more interesting info there. Notice the Recycle column? We’ve forbidden Bacula to recycle old volumes when we defined the pool resource. We can fix that, right? Let’s exit the bconsole and edit the configuration file:

* exit
# vi includes/dir_pool.conf

Change the respective line to:

Recycle = yes

Then save the file and exit the editor. The configuration changed and so the dir needs to reinitialize. To do so we restart it.

# service bacula-dir restart

Seems like it’s not responding? Hit CTRL+C to cancel. Let’s stop and start it instead:

# service bacula-dir stop
# service bacula-dir start

That worked. Now we enter the bconsole again and take a look at the volumes:

# bconsole
* list volumes

Huh? That configuration change didn’t work! The recycling flag is still set to 0. Why? There is an easy answer to that: Because this value is not read from the configuration! It comes from the catalog. The configuration setting is applied at the moment a new value is created. Once the volume exists, the configuration setting is irrelevant. Of course we are not out of luck here. We can modify the flag using the bconsole (and yes, that asterisk right before the MediaId (the three in this case) is NOT a prompt symbol; type it in!):

* update
1
7
4
*3
* yes
18

Let’s see if that worked:

* list volumes

It did! So will Bacula now reuse the old volume and overwrite all data on it? No it won’t. Bacula knows that there’s data on it because it keeps track of all that in the catalog. And it tries to preserve that data even though the volume allows recycling.

Purging a volume

However we can tell Bacula to get rid of the catalog data that references this volume. To do so, we purge job and volume information for that volume from the catalog:

purge [purge jobs volume]
3
4
*3
* list volumes

[…]
Purged
Full
Full
[…]

See how the VolStatus changed? It’s purged and recycling is enabled. That means that Bacula will reuse the volume. But will our job resume automatically? Let’s take a look at it:

* list jobs

Oh no! What’s that JobStatus? It’s “f” for failed! What happened? Well, we stopped the director, remember? That killed the running job! So to try out recycling we need to run another backup job. Let’s start the job now:

* run
4
yes

Take a look at the volumes again:

[…]
Recycle
Full
Full
[…]

The VolStatus changed to Recycle and Bacula will reuse the old volume. In theory we’d have to purge one more volume for our backup to succeed. But since this is just an example job to show off some important things, we’re actually done at this point. But for today’s tasks we’re done now.

Save the current status for later:

* exit
# shutdown -p now
% vagrant status
% vagrant snapshot save tut_5_end

Intermission

You now know the basics of pool creation and some of the features that come with it. You’ve also purged and recycled a volume and should have a better understanding of how Bacula works in general. There’s a lot more to pools however and the next post in this series should probably go into retention periods and discuss a topic that we’ve only touched upon so far: The Catalog.

However this post concludes my „Bacula October“ and I’ll end this tutorial series here. It takes a lot of effort and time to write these posts and while I hope that this is of any use to somebody, I have no idea whether it is or not. For that reason I might or might not take this topic up again in the future. I had planned to simulate multiple backup clients with vagrant, do encrypted backups and so on. But now I’m looking forward to write about something else again! Of course feel free to comment on any of the parts if you liked (or want to tell my why you didn’t like) this tutorial.