Multi-OS PXE-booting from FreeBSD 12: PXE menu and *BSD (pt. 3)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/multi-os_pxe-booting_from_fbsd_pt3.gmi

Post 1 of this mini series is about what lead me to do this in the first place, features a little excursion for people new to PXE and details the setup of a FreeBSD router.
Post 2 discusses setting up the required daemons for DHCP, TFTP and HTTP / FTP. Each component that is not in FreeBSD’s base system is briefly discussed and two options to pick from are covered.

At the end of part 2, the situation is as follows: A client in the 10.11.12.0/24 subnet attempting to PXE boot will get an IP address via DHCP and be told where to get the NPB (Network Bootstrap Program). It will then attempt to fetch it via TFTP. There’s just one problem: That is not there, yet! We’ll fix that in a minute. When we’re done here, you’ll have a fully working PXE server that offers multiple BSD operating systems to boot into (Linux and more are covered in part 4). This article focuses on BIOS (“legacy”) booting; if you want to boot EFI-only machines you’ll have to adapt the configuration examples given here to that. I also assume using HTTP here – if you opted for FTP you will have to adapt the commands used in the examples.

Network Bootstrap Program

There are a lot of NBPs available. Usually each operating system has its own which is tuned towards its specific duty: FreeBSD has one, OpenBSD has another and Linux has several. These are not ordinary programs; they need to cope with a very resource-constrained environment and cannot depend on any external libraries. While writing boot code is challenging enough, adding network booting capabilities doesn’t make things any easier. This is why most NBPs are as simple as possible.

As a result of that, the NBPs usually know how to boot exactly one operating system. Since we want to set up a multi-OS PXE server this is quite unfortunate for our use case. There are two ways to work around this problem:

  1. Provide various NBPs and use DHCP to distinguish between clients
  2. Use an NBP that supports a menu to select which one to boot next

As usual there’s pros and cons to both. Letting DHCP do the magic requires a much more complex DHCP configuration. It’s also much less flexible. The boot menu approach is simple and flexible, but more complicated if you are also interested in automation. I do like automation, but I decided in favor of using a boot menu for this article because it’s easier to follow. It is also a good achievement to build upon once you’re comfortable with DHCP and feel ready for advanced settings.

It is possible to use one NBP to fetch and execute another one. This process is known as chainloading. For some operating systems that is the best choice to add them to a menu system.

There’s three popular options that we have for an NBP which fits our needs:

1. GRUB
2. PXELINUX (from Syslinux) and
3. iPXE

GRUB and I never made friends. I used it for a while after switching from LILO only to ditch it for Syslinux when I learned of that. I have to use it on many systems, but when I have a choice, I choose something else. The iPXE project is very interesting. It’s the most advanced (but also most involved) of the three options. If you’re curious about just how far you can take PXE booting, at least have a look at it. For this article, we’ll go with PXELINUX.

Pxelinux

Pxelinux is available via packages on FreeBSD. It does pull in some dependencies that I don’t want on my system however. For that reason we’re going to fetch the package instead of installing it. Then we extract it manually:

# pkg fetch -y syslinux
# mkdir /tmp/syslinux
# tar -C /tmp/syslinux -xvf /var/cache/pkg/syslinux-6.03.txz

Now we can cherry-pick the required files:

# cp /tmp/syslinux/usr/local/share/syslinux/bios/core/lpxelinux.0 /usr/local/tftpboot/pxelinux.0
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/elflink/ldlinux/ldlinux.c32  /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/menu/vesamenu.c32 /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/lib/libcom32.c32 /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/libutil/libutil.c32 /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/modules/pxechn.c32 /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/memdisk/memdisk /usr/local/tftpboot/
# rm -r /tmp/syslinux

The first one is the modular NBP itself. It requires some modules – the c32 files. The pxechn and memdisk modules are optional – they are required for some of the operating systems examples here but not all. You can leave them out if you don’t need them. Restart the inetd service now and you will be able to PXE boot to the menu:

# service inetd restart

Keep in mind: Restart the inetd service whenever you added a file to or edited any in the tftpboot directory!

Tip 1: You can use make use of gzipped files as Pxelinux supports that. This way you can decrease loading times by serving smaller images over the net.

Tip 2: I’m using gzip in my examples but if you really want to fight for the last byte, use zopfli instead. It’s a compression program that produces gzip-compatible output but takes much more processor time to create optimized archives. As decompression time is unaffected it’s a price you have to pay only once. Consider using it if you want the best results.

Submenus

Pxelinux is hard-coded to load pxelinux.cfg/default via TFTP and use that as its configuration. If you plan to only use few of the OS examples shown here, that config is sufficient as you can put everything into there. Once you feel that your menu is becoming overly crowded, you can turn it into a main menu and make use of submenus to group things as I do it here. This works by putting the various OS entries in different config files.

If you don’t want submenus, skip the next step and put all the menu entries that go into something other than pxelinux.cfg/default in my examples right into that file instead – and leave out the reference back to the main menu since they don’t make any sense if you’re using a flat menu anyway.

In the previous post we already created the file /usr/local/tftpboot/pxelinux.cfg/default. Append the following to it to create a submenu for the BSDs we’re covering today:

MENU TITLE PXE Boot Menu (Main)

LABEL bsd-oses
        MENU LABEL BSD Operating Systems
        KERNEL vesamenu.c32
        APPEND pxelinux.cfg/bsd

Now create the file referenced there:

# vi /usr/local/tftpboot/pxelinux.cfg/bsd

and put the following text in there:

MENU TITLE PXE Boot Menu (BSD)

LABEL main-menu
        MENU LABEL Main Menu
        KERNEL vesamenu.c32
        APPEND pxelinux.cfg/default

Alright, preparation work is done, let’s finally add some operating system data!

FreeBSD 12.2

PXE booting FreeBSD is actually not such an easy thing to do if you want to avoid using NFS shares. Fortunately there is mfsBSD, a project that provides tools as well as releases of special FreeBSD versions that can be booted over the net easily. We’re going to use that.

There are multiple variants for download: The standard one, the “special edition” and a “mini edition”. The special edition comes with the distribution tarballs on the ISO – you may want to use that one for installation purposes. If you just want a FreeBSD live system (e.g. for maintenance and repairs) or use you owr mirror (see below), the standard edition is for you since it is much smaller and thus boots way faster.

Let’s make the image available via HTTP:

# mkdir -p /usr/local/www/pxe/bsd/fbsd
# fetch https://mfsbsd.vx.sk/files/iso/12/amd64/mfsbsd-12.2-RELEASE-amd64.iso -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso
# gzip -9 /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso

Now edit the pxelinux.cfg/bsd file and append:

LABEL fbsd-pxe-install
        MENU LABEL Install FreeBSD 12.2 (PXE)
        MENU DEFAULT
        KERNEL memdisk
        INITRD http://10.11.12.1/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso
        APPEND iso raw

That’s it. You can now PXE-boot into FreeBSD.

Installation using mfsBSD

Login with user root and password mfsroot. It includes a “zfsinstall” script that you may want to take a look at. There’s a lot more to mfsBSD, though. Its tools allow you to easily roll your own customized images. If a way to include packages or files, use a custom-built kernel and things like that sounds like something that would be useful for you, take a closer look. I cannot go into more detail here – it’s a topic of its own and would deserve an entire article dedicated to it. In case you just want to use the familiar FreeBSD installer bsdinstall, read on.

Mirroring distfiles and fixing bsdinstall

If you want to install FreeBSD over PXE more than once, it makes sense to provide a local distfile mirror. Since we have a fileserver running anyway, there’s really nothing more to it than getting the distfiles and putting them into the right place. At the very minimum get the following three files:

# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/MANIFEST -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/MANIFEST
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/base.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/base.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/kernel.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/kernel.txz

Depending on which distfiles you usually install, also get any or all of the following files:

# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/base-dbg.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/base-dbg.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/kernel-dbg.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/kernel-dbg.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/lib32-dbg.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/lib32-dbg.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/ports.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/ports.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/src.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/src.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/tests.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/tests.txz

At this point in time, bsdinstall is broken on mfsBSD. The reason is that the distfile manifest is missing. I think about getting this fixed upstream, so in the future try and see if the following part is obsolete before using it. But for now, let’s create a simple shell script in the webroot directory for convenience:

# vi /usr/local/www/pxe/fbsd.sh

Put the following into the script:

#!/bin/sh
ARCH=`uname -m`
RELEASE=`uname -r | cut -d"-" -f1`
mkdir -p /usr/freebsd-dist
fetch http://10.11.12.1/bsd/fbsd/${ARCH}/${RELEASE}-RELEASE/MANIFEST -o /usr/freebsd-dist/MANIFEST
bsdinstall

Now if you PXE-booted mfsBSD and logged in as root, you just need to execute the following command line and will then be able to use the installer as you are used to it:

# fetch http://10.11.12.1/fbsd.sh && sh fbsd.sh

When you are to select the installation source, there is an “Other” button at the bottom of the window. Choose that and point to your distfile mirror – in my example http://10.11.12.1/bsd/fbsd/amd64/12.2-RELEASE. Happy installing!

One more hint: You may want to look into the environment variables that bsdinstall(8) provides. I briefly attempted to automatically set the URL to the distfile mirror but couldn’t get it working. As I was already running out of time with this article I haven’t looked deeper into it. If anybody figures it out I’d appreciate sharing your solution here.

OpenBSD 6.8

Adding OpenBSD as an option is trivial. The project provides a ramdisk kernel used for installing the system and a NBP capable of loading it. Let’s get those two files in place – and while the ramdisk kernel is fairly small already, we can chop a couple of bytes off by compressing it:

# mkdir -p /usr/local/tftpboot/bsd/obsd
# fetch https://cdn.openbsd.org/pub/OpenBSD/6.8/amd64/pxeboot -o /usr/local/tftpboot/bsd/obsd/pxeboot
# fetch https://cdn.openbsd.org/pub/OpenBSD/6.8/amd64/bsd.rd -o /usr/local/tftpboot/bsd/obsd/6.8-amd64.rd
# gzip -9 /usr/local/tftpboot/bsd/obsd/6.8-amd64.rd

Now we only need to add the required lines to pxelinux.cfg/bsd:

LABEL obsd-pxe-install
        MENU LABEL Install OpenBSD 6.8 (PXE)
        KERNEL pxechn.c32
        APPEND bsd/obsd/pxeboot

That’s it, the OpenBSD loader can be booted! Since we don’t have the kernel in the assumed default location (“/bsd”) we’d need to tell the loader to “boot bsd/obsd/6.8-amd64.rd.gz”. The loader supports a configuration file, though. So for a little extra convenience we can make it pick up the kernel automatically like this:

# mkdir -p /usr/local/tftpboot/etc
# ln -s /usr/local/tftpboot/etc/boot.conf /usr/local/tftpboot/bsd/obsd/boot.conf
# echo "boot bsd/obsd/6.8-amd64.rd.gz" > /usr/local/tftpboot/bsd/obsd/boot.conf
# echo "# OpenBSD boot configuration" >> /usr/local/tftpboot/bsd/obsd/boot.conf

The pxeboot program comes with the configuraton file name of etc/boot.conf hard-coded. To keep things a little cleaner in the hierarchy that I use, I chose to set a symlink in the obsd directory for reference purposes. And that’s all.

NetBSD 9.1

Let’s add NetBSD! It’s somewhat similar to OpenBSD – but a bit more involved unfortunately. The reason is that the NBP by default does not support a configuration file. It has the ability to use one, but that needs to be activated first. Which is fair enough since it’s only a single command – on NetBSD that is! But let’s worry about this in a minute and first get the NBP as well as the install kernel:

# mkdir -p /usr/local/tftpboot/bsd/nbsd
# fetch https://cdn.netbsd.org/pub/NetBSD/NetBSD-9.1/amd64/installation/misc/pxeboot_ia32.bin -o /usr/local/tftpboot/bsd/nbsd/pxeboot_ia32.bin
# fetch https://cdn.netbsd.org/pub/NetBSD/NetBSD-9.1/amd64/binary/kernel/netbsd-INSTALL.gz -o /usr/local/tftpboot/bsd/nbsd/netbsd-INSTALL.gz

Now we need to add the boot menu entry by adding the following lines to pxelinux.cfg/bsd:

LABEL nbsd-pxe-install
        MENU LABEL Install NetBSD 9.1 (PXE)
        KERNEL pxechn.c32
        APPEND bsd/nbsd/pxeboot_ia32.bin

This is sufficient to load and execute the NetBSD loader. That will then complain that it cannot find the kernel and no hints about NFS were given. Now we have three options:

  1. Manually point the loader to the correct kernel each time
  2. Give the required hint via DHCP
  3. Try to enable the loader configuration

Typing in “tftp:bsd/nbsd/netbsd-INSTALL.gz” is probably fair enough if you are doing NetBSD installs very rarely but it gets old quickly. So let’s try out option two!

Modifying DHCP config for NetBSD

The DHCP server needs to be configured to pass a different Boot File name option when answering the NetBSD loader than otherwise. This is done by matching class information. This topic is beyond the scope of this article, so if you are interested, do some reading on your own. I won’t leave you hanging, though, if you just need to get things working.

Here’s what you have to add to the configuration if you’re using Kea – for example right before the “loggers” section:

    "client-classes": [
        {
            "name": "NetBSDloader",
            "test": "option[vendor-class-identifier].text == 'NetBSD:i386:libsa'",
            "boot-file-name": "tftp:bsd/nbsd/netbsd-INSTALL.gz"
        }
    ],

And here the same thing if you are using DHCPd:

if substring (option vendor-class-identifier, 0, 17) = "NetBSD:i386:libsa" {
    if filename = "netbsd" {
        filename "tftp:bsd/nbsd/netbsd-INSTALL.gz";
    }
}

Restart your DHCP server and you should be good to go.

After accomplishing the easy way via DHCP, I also went down to explore the boot.cfg road but ultimately failed. I’m documenting it here anyway in case somebody wants to pick up where I decided to leave it be.

Enabling boot.cfg in the loader

To mitigate the risk of polluting my main system by doing something stupid I chose to do all of this using my unprivileged user. The first thing I did, was fetching and extracting the basic NetBSD 9.1 sources:

% mkdir -p netbsd-9.1 && cd netbsd-9.1
% fetch ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-9.1/source/sets/src.tgz
% tar xvzf src.tgz

The sources for the installboot program we’re looking for are in usr/src/usr.sbin/installboot. I tried to get that thing to build by pointing the compiler at additional include directories and editing quite some header files, hoping to resolve the conflicts with FreeBSD’s system headers and problems like that. It can probably be done but that would take a C programmer – which I am not.

Fortunately NetBSD is the portability star among the BSDs and should be buildable on many other systems. I’ve never done this before but here was the chance. So I installed CVS and checked out the rest of the NetBSD src module:

% doas pkg install -y cvs
% cd netbsd-9.1/usr
% cvs -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -r netbsd-9-1-RELEASE -P src

When exploring the source tree, I found a build script that obviously does all the magic required here. Be warned however that this builds a full cross-toolchain including the complete GCC compiler! Then it builds the “tools” subset of the NetBSD code (which includes the installboot that we’re looking for). On my old and slow Atom-based system this process took 6 hours:

% cd src
% ./build.sh -U -m amd64 -T ~/nbsd tools
===> build.sh command:    ./build.sh -U -m amd64 -T /home/kraileth/nbsd tools
===> build.sh started:    Thu Feb  3 00:13:41 CET 2021
===> NetBSD version:      9.1
===> MACHINE:             amd64
===> MACHINE_ARCH:        amd64
===> Build platform:      FreeBSD 12.2-RELEASE-p1 amd64
===> HOST_SH:             /bin/sh
===> No $TOOLDIR/bin/nbmake, needs building.
===> Bootstrapping nbmake
checking for sh... /bin/sh
checking for gcc... cc

[...]

install ===> config
#   install  /home/kraileth/nbsd/bin/nbconfig
mkdir -p /home/kraileth/nbsd/bin
/home/kraileth/nbsd/bin/amd64--netbsdelf-install -c  -r -m 555 config /home/kraileth/nbsd/bin/nbconfig
===> Tools built to /home/kraileth/nbsd
===> build.sh ended:      Thu Feb  3 6:26:25 CET 2021
===> Summary of results:
         build.sh command:    ./build.sh -U -m amd64 -T /home/kraileth/nbsd tools
         build.sh started:    Thu Feb  4 07:13:41 CET 2021
         NetBSD version:      9.1
         MACHINE:             amd64
         MACHINE_ARCH:        amd64
         Build platform:      FreeBSD 12.2-RELEASE-p1 amd64
         HOST_SH:             /bin/sh
         No $TOOLDIR/bin/nbmake, needs building.
         Bootstrapping nbmake
         MAKECONF file:       /etc/mk.conf (File not found)
         TOOLDIR path:        /home/kraileth/nbsd
         DESTDIR path:        /usr/home/kraileth/netbsd-9.1/usr/src/obj/destdir.amd64
         RELEASEDIR path:     /usr/home/kraileth/netbsd-9.1/usr/src/obj/releasedir
         Created /home/kraileth/nbsd/bin/nbmake
         Updated makewrapper: /home/kraileth/nbsd/bin/nbmake-amd64
         Tools built to /home/kraileth/nbsd
         build.sh ended:      Thu Feb  4 12:26:25 CET 2021
===> .

The “-U” flag enables some trickery to build as an unprivileged user. With “-m” you specify the target architecture (I did use i386 but modified the above lines as that will be what most people will want to use instead!). Finally the “-T” switch allows to specify the installation target directory and the “tools” is the make target to use.

When it was done, I did the following (as root):

# cp /usr/local/tftpboot/bsd/nbsd/pxeboot_ia32.bin /usr/local/tftpboot/bsd/nbsd/pxeboot_ia32.bin.bak
# /usr/home/kraileth/netbsd-9.1/usr/src/tools/installboot/obj/installboot -eo bootconf /usr/local/tftpboot/bsd/nbsd/pxeboot_ia32.bin

This should enable the boot config file on the pxeboot loader. It does change the file and probably even makes the right change. I tried to enable module support via installboot, too and that obviously worked (the NFS module was loaded the next time I chainloaded the NetBSD loader). But for some reason I could not get boot.cfg to do what I wanted. Probably I don’t understand the file properly…

While it’s a bit disappointing to stop so close to the goal, messing with NetBSD so much already took much more time away from the other BSDs than I had imagined. And since I could at least offer a working way this was when I decided to move on.

DragonFly BSD

I attempted to get DragonFly BSD to work but failed. I briefly tried out a setup that includes NFS shares but it didn’t work completely either: Kernel booted but failed to execute /sbin/init for some reason or another. Also I don’t really want to cover NFS in this series – there’s enough material in here already. And without NFS… Well, DragonFly BSD has the same problem that FreeBSD has: It will boot the kernel but then be unable to mount the root filesystem.

While I guess that the mfsBSD approach could likely work for DF, too, this is something much more involved than reasonable for our topic here. I would really like to cover DragonFly here, too, but that’s simply a bit too much. If anybody knows how to get it working – please share your knowledge!

HardenedBSD 12-STABLE

HardenedBSD being a FreeBSD fork, inherited the same characteristics as vanilla FreeBSD. Which means that PXE booting the standard images is not an easy thing to do. HardenedBSD uses the same installer, bsdinstall however and for that reason it’s possible to install HardenedBSD by using mfsBSD as prepared above in the FreeBSD section. We only need to point the installer to a different distfile mirror. Let’s create that one now:

# mkdir -p /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE
# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/MANIFEST -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/MANIFEST
# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/base.txz -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/base.txz
# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/kernel.txz -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/kernel.txz

As with FreeBSD, there are some optional distfiles you may or may not want to mirror, too. Provide what you need for your installations:

# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/base-dbg.txz -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/base-dbg.txz
# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/kernel-dbg.txz -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/kernel-dbg.txz
# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/src.txz -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/src.txz
# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/tests.txz -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/tests.txz

Now we’re creating a convenience script for HardenedBSD:

# vi /usr/local/www/pxe/hbsd.sh

Put the following into the script:

#!/bin/sh
ARCH=`uname -m`
MAJOR=`uname -r | cut -d"." -f1`
mkdir -p /usr/freebsd-dist
fetch http://10.11.12.1/bsd/hbsd/${ARCH}/${MAJOR}-STABLE/MANIFEST -o /usr/freebsd-dist/MANIFEST
bsdinstall

Now fire up mfsBSD, login as root and simply run the following command line to start the installer:

# fetch http://10.11.12.1/hbsd.sh && sh hbsd.sh

Select the “Other” button when asked for the installation source. Choose that and point to your distfile mirror – in my example http://10.11.12.1/bsd/hbsd/amd64/12-STABLE. And that’s it.

MidnightBSD 2.0

Since MidnightBSD is a FreeBSD fork as well, it also suffers from the well-known problems related to PXE booting. Again mfsBSD comes to the rescue. Let’s create the distfile mirror first:

# mkdir -p /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/MANIFEST -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/MANIFEST
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/base.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/base.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/kernel.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/kernel.txz

Pick any or all of the remaining optional distfiles to be mirrored, too, if you need them:

# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/lib32.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/lib32.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/doc.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/doc.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/base-dbg.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/base-dbg.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/kernel-dbg.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/kernel-dbg.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/lib32-dbg.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/lib32-dbg.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/src.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/src.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/mports.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/mports.txz

And here’s the convenience script:

# vi /usr/local/www/pxe/mbsd.sh

Put the following into it:

#!/bin/sh
ARCH=`uname -m`
RELEASE="2.0"
mkdir -p /usr/freebsd-dist
fetch http://10.11.12.1/bsd/mbsd/${ARCH}/${RELEASE}-RELEASE/MANIFEST -o /usr/freebsd-dist/MANIFEST
bsdinstall

Now you can PXE-boot mfsBSD as prepared in the FreeBSD section. After logging in as root execute the following command line that will take you to the installer:

# fetch http://10.11.12.1/mbsd.sh && sh mbsd.sh

When given the choice to select the installation source make sure to select “Other” and point to the right URL – in my example it would be http://10.11.12.1/bsd/mbsd/amd64/2.0-RELEASE, then install the OS as you’re used to.

Watch out for some options in the installer, though! MidnightBSD is mostly on par with FreeBSD 11.4. If you enable e.g. newer hardening options that the installer knows but the target OS doesn’t, you might run into trouble (not sure, tough, I didn’t think of this until after my test installation).

What’s next?

I didn’t plan this post to get that long, especially NetBSD surprised me. Eventually I decided to cover HardenedBSD and MidnightBSD as well and accept that this is a BSD-only article. So the next one will add various Linuxen and other operating systems to the mix.

Multi-OS PXE-booting from FreeBSD 12: Required services (pt. 2)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/multi-os_pxe-booting_from_fbsd_pt2.gmi

The previous post was about what lead me to do this in the first place, featured a little excursion for people new to PXE and most importantly detailed the setup of a FreeBSD router that will be turned into a PXE server in this article.

While I originally intended to show how to boot Ubuntu in this part, things have changed a little. I realized that the software choices I made might not be what a lot of people would have chosen. Therefore I did a little extra work and present my readers with multiple options. This made the article long enough even without the Ubuntu bits which I wanted to cover in part part 3 instead (it was moved to part 4, however).

Current state and scope

Today we will make the machine offer all the services needed for network installations of many Open Source operating systems. My examples are all IPv4, but it should be possible to adapt this to IPv6 fairly easily. As this is *nix, there’s always multiple ways to do things. For software that does not come included in FreeBSD’s base installation, I’ll be giving two options. One will be less commonly deployed but have some advantages in my book. The other will achieve the same goal with a more popular solution.

The machine that I use is an old piece of metal that has 6 NICs – which is why I like to still use it for tinkering with network-related stuff. After part 1, we left off with a gateway that has an Internet-facing adapter em0 which gets its IP via DHCP from my actual router. We’re also using em5 which is statically configured to have the IP 10.11.12.1 and is connected to a separate switch. There’s an unbound nameserver running and serving that interface and a pf firewall is active doing NAT.

This means that everything is in place to serve any host connected to said switch, if it’s manually configured to use a static address in the 10.11.12.0/24 IP range and with default router and nameserver set to 10.11.12.1. Let’s start by getting rid of that “manually configured” requirement!

Excursion: DHCP basics

We do so by configuring the machine as a DHCP server handing out IP addresses for the 10.11.12.0/24 subnet on the 6th NIC. DHCP servers work by listening for DHCP requests which are broadcasted on the network (as the client does not have it’s own IP, yet). When receiving one, it will have a look at its configuration: Is there anything special for the host asking? Is the MAC address included with the request maybe blacklisted? Is there a reserved IP to be handed to this specific machine? Or any particular option to send to the “class” of device asking?

In our simple case there aren’t really any bells and whistles involved. So it will have a look at the IP pool that it manages and if it can find an unused one, it will answer with a DHCP offer. The latter is a proposal for an IP to be leased by the client and also includes various network-related information: Usually at least the netmask, router and nameserver. A lot of additional information can be provided in form of options; you can point the client joining the network to a time server if you have one, inform about the domain being used and much, much more (even custom options are possible if you need them).

For PXE booting to work we need to make use of two particular options: We need the PXE code in the firmware to know which server to turn to if it wants to load the NBP (Network Bootstrap Program) from the net. It also needs to know what the file to ask for is called.

DHCP servers

There are multiple DHCP servers out there. If I were doing this on Linux, I’d probably just pick Dnsmasq and be done: As the name implies, it does DNS. But it also does DHCP and TFTP (which we are in need of as well) and supports PXE. But FreeBSD comes with its own TFTP daemon that I’m going to use and I actually prefer the Unix way over all-rounder software: Do one thing and do it well!

The first thing that comes to mind in terms of DHCP servers is ISC’s DHCPd. It’s small, simple to use (at least for our use case), battle-tested and in use pretty much everywhere. It’s old, though, not extremely fast and certainly not as flexible as you might wish for today. This (among other things) lead the ISC to start a new project meant as a replacement: Kea.

The latter is a modern DHCP server with a lot of nice new features: It is a high-performance solution that’s extensible, supports databases as backends, has a web GUI (Stork) available and more. But since DHCPd works well enough, adoption of Kea has been slow. There are a couple of downsides to it, too: First and foremost – its configuration is written in JSON. Yes, JSON! While there are legitimate use cases for that format, configuration is not one of them if you ask me. That was a terrible choice. Kea also pulls in big dependencies like the boost C++ libraries not everybody is fond of.

IMO the benefits of Kea outweight the drawbacks (if it wasn’t for the JSON configuration, I’d even state: clearly). But it’s your choice of course.

DHCP server option 1: Modern-day Kea

Alright, let’s give Kea a try, shall we? First we need to install it and then edit the configuration file:

# pkg install -y kea
# vi /usr/local/etc/kea/kea-dhcp4.conf

The easiest thing to do is to delete the contents and paste the following. Then adapt it to your network and save:

{
"Dhcp4": {
    "interfaces-config": {
        "interfaces": [ "em5/10.11.12.1" ]
    },
    "control-socket": {
        "socket-type": "unix",
        "socket-name": "/tmp/kea4-ctrl-socket"
    },
    "lease-database": {
        "type": "memfile",
        "lfc-interval": 3600
    },
    "expired-leases-processing": {
        "reclaim-timer-wait-time": 10,
        "flush-reclaimed-timer-wait-time": 25,
        "hold-reclaimed-time": 3600,
        "max-reclaim-leases": 100,
        "max-reclaim-time": 250,
        "unwarned-reclaim-cycles": 5
    },

    "renew-timer": 900,
    "rebind-timer": 1800,
    "valid-lifetime": 3600,
    "next-server": "10.11.12.1",
    "boot-file-name": "pxelinux.0",

    "option-data": [
        {
            "name": "subnet-mask",
            "data": "255.255.255.0"
        },
        {
            "name": "domain-name-servers",
            "data": "10.11.12.1"
        },
        {
            "name": "domain-name",
            "data": "example.org"
        },
        {
            "name": "domain-search",
            "data": "example.org"
        }
    ],

    "subnet4": [
        {
            "subnet": "10.11.12.0/24",
            "pools": [ { "pool": "10.11.12.20 - 10.11.12.30" } ],
            "option-data": [
                {
                    "name": "routers",
                    "data": "10.11.12.1"
                }
            ]
        }
    ],

    "loggers": [
    {
        "name": "kea-dhcp4",
        "output_options": [
            {
                "output": "/var/log/kea-dhcp4.log"

            }
        ],
        "severity": "INFO",
        "debuglevel": 0
    }
  ]
}
}

Yes, looks pretty bad, I know. But that’s only the representation; if something better had been used (say YAML), it’d be about 50 lines instead of 75, be much more readable and above all: less error-prone to edit. Oh well. If you can ignore the terrible representation, the actual data is not so bad and pretty much self-explaining.

I’d like to point you at the “next-server” and “boot-file-name” global options that I set here. These are required for PXE booting by pointing to the server hosting the NBP and telling its file name. Leave them out and you will still have a working DHCP server if you don’t need to do PXE. While this configuration works, you will likely want to extend it for production use.

With the config in place, let’s enable and start the daemon:

# sysrc kea_enable="YES"
# service kea start

A quick look if a daemon successfully bound to port 67 and is listening doesn’t hurt:

# sockstat -4l | grep 67
root     kea-dhcp4  1480  14 udp4   10.11.12.1:67         *:*

Ok, there we are. We now have a DHCP service on our internal network!

DHCP server option 2: Venerable ISC DHCPd

So you’d like to play simple and safe? No problem, going with DHCPd is not a bad choice. But first we need to install it and edit the configuration file:

# pkg install -y isc-dhcp44-server
# vi /usr/local/etc/dhcpd.conf

Delete everything. Then add the following (adjust to your network structure, of course!) and save:

option domain-name "example.org";
option domain-name-servers 10.11.12.1;
option subnet-mask 255.255.255.0;
 
default-lease-time 600;
max-lease-time 7200;
ddns-update-style none;
log-facility local7;

next-server 10.11.12.1;
filename "pxelinux.0";
 
subnet 10.11.12.0 netmask 255.255.255.0 {
  range 10.11.12.10 10.11.12.20;
  option routers 10.11.12.1;
}

Mind the “next-server” and “filename” options which define the server to get the NBP from as well as the file name of that. You can leave out that block and will still have a working DHCP server – but it won’t allow for PXE booting in that case. I’d also advice you to do a bit of reading and probably do a more comprehensive configuration of DHCPd.

Next thing to do is to enable DHCPd, confine it to serve requests coming in from one particular NIC only and start the service:

# sysrc dhcpd_enable="YES"
# sysrc dhcpd_ifaces="em5"
# service isc-dhcpd start

Quick check to see if the service is running and binding on port 67:

# sockstat -4l | grep 67
dhcpd    dhcpd      1396  8  udp4   *:67                  *:*

Looking good so far, DHCP should be available on our internal net.

Optional: Checking DHCP

If you want to make sure that your DHCP server is not only running but that it can also be reached and actually does what it’s meant to, you can either just try to power up a host in the 10.11.12.0/24 network and configure it to get its IP via DHCP. Or you can for example use the versatile nmap tool to test DHCP discovery from any host on that network:

# pkg install -y nmap
# nmap --script broadcast-dhcp-discover
Starting Nmap 7.91 ( https://nmap.org ) at 2021-01-24 17:56 CET
Pre-scan script results:
| broadcast-dhcp-discover: 
|   Response 1 of 1: 
|     Interface: em0
|     IP Offered: 10.11.12.21
|     DHCP Message Type: DHCPOFFER
|     Subnet Mask: 255.255.255.0
|     Router: 10.11.12.1
|     Domain Name Server: 10.11.12.1
|     Domain Name: example.org
|     IP Address Lease Time: 1h00m00s
|     Server Identifier: 10.11.12.1
|     Renewal Time Value: 15m00s
|_    Rebinding Time Value: 30m00s
WARNING: No targets were specified, so 0 hosts scanned.
Nmap done: 0 IP addresses (0 hosts up) scanned in 10.56 seconds

# pkg delete -y nmap

All right! DHCP server is working and happily handing out leases.

TFTP

The Trivial File Transfer Protocol daemon is up next. FreeBSD ships with a TFTP daemon in the base system, so we’re going to use that. It will not be used by itself but instead from the inetd super daemon. To enable TFTP, we just need to put one line in the inetd configuration file:

# echo "tftp    dgram   udp     wait    root    /usr/libexec/tftpd      tftpd -l -s /usr/local/tftpboot" >> /etc/inetd.conf

Now we need to create the directory that we just referenced, as well as a subdirectory which we’re going to use and create a file there:

# mkdir -p /usr/local/tftpboot/pxelinux.cfg
# vi /usr/local/tftpboot/pxelinux.cfg/default

Put the following in there (for now) and save:

DEFAULT vesamenu.c32
PROMPT 0

All is set, let’s enable and start the service now:

# sysrc inetd_enable="YES"
# service inetd start

Again we can check real quick if the service is running:

# sockstat -4l | grep 69
root     inetd      1709  6  udp4   *:69                  *:*

Good, so now we can test to fetch the configuration file from either our server or from any FreeBSD machine in the 10.11.12.0/24 network:

# tftp 10.11.12.1
tftp> get pxelinux.cfg/default
Received 292 bytes during 0.0 seconds in 1 blocks
tftp> quit
# rm default

Everything’s fine just as expected.

File Server

The remaining piece we need to set up is a means to efficiently transfer larger files over the wire – i.e. not TFTP! You can do it via FTP and use FreeBSD’s built-in FTP daemon. While this works well, it is not the option that I’d recommend. Why? Because FTP is an old protocol that does not play nicely with firewalls. Sure, it’s possible to do FTP properly, but that’s more complex to do than using something else like a webserver that speaks HTTP.

If you want to go down that path, there are a lot of options. There’s the very popular and feature-rich Apache HTTPd and various more light-weight solutions like LighTTPd and many more. I generally prefer OpenBSD’s HTTPd because it is so easy to work with – and when it comes to security or resisting feature creep its developers really mean it. If I need to do something that it cannot do, I usually resort to the way more advanced (and much more popular) Nginx.

Pick any of the two described here, go with FTPd instead or just ignore the following three sections and set up the webserver that you prefer to deploy.

If you didn’t opt for FTP, as a first step create a directory for the webserver and change the ownership:

# mkdir -p /usr/local/www/pxe
# chown -R www:www /usr/local/www/pxe

File Server option 1: OpenBSD’s HTTPd

Next is installing the program and providing the desired configuration. Edit the file:

# pkg install -y obhttpd
# vi /usr/local/etc/obhttpd.conf

Delete the contents and replace it with the following, then save:

chroot "/usr/local/www"
logdir "/var/log/obhttpd"
 
server "pixie.example.org" {
        listen on 10.11.12.1 port 80
        root "/pxe"
        log style combined
}

Super simple, eh? That’s part of the beauty of obhttpd. OpenBSD follows the “sane defaults” paradigm. That way you only have to configure stuff that is specific to your task as well as things where you want to change the defaults. Surprisingly, this configuration does it – and there’s really not much I’d change for a production setup if it is the only site on this server.

It’s always a good idea to check if the configuration is valid, so let’s do that:

# obhttpd -nf /usr/local/etc/obhttpd.conf
configuration OK

If you ever need to debug something, you can start the daemon in foreground and more verbosely by running obhttpd -dvv. Right now the server would not start because the configured log directory does not exist. So this would be a chance to give debugging a try.

Let’s create the missing directory and then enable and start the service:

# mkdir /var/log/obhttpd
# sysrc obhttpd_enable="YES"
# service obhttpd start

As always I prefer to take a quick look if the daemon did bind the way I wanted it to:

# sockstat -4l | grep httpd
www      obhttpd    1933  7  tcp4   10.11.12.1:80         *:*
www      obhttpd    1932  7  tcp4   10.11.12.1:80         *:*
www      obhttpd    1931  7  tcp4   10.11.12.1:80         *:*

Looks good.

File Server option 2: Nginx

Next thing is installing Nginx and providing a working configuration:

# pkg install -y nginx
# vi /usr/local/etc/nginx/nginx.conf

Erase the example content and paste in the following:

user  www;
error_log  /var/log/nginx/error.log;
worker_processes  auto;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    keepalive_timeout  65;

    server {
        listen       80;
        location / {
            root   /usr/local/www/pxe;
        }
    }
}

This is by no means a perfect configuration but only an example. If you want to deploy Nginx in production, you’ll have to further tune it towards what you want to achieve. But now let’s enable and start the daemon:

# sysrc nginx_enable="YES"
# service nginx start

Time for the usual quick check:

# sockstat -4l | grep nginx
www      nginx      1733  6  tcp4   *:80                  *:*
www      nginx      1732  6  tcp4   *:80                  *:*
root     nginx      1731  6  tcp4   *:80                  *:*

Nginx is running and listening on port 80 as it should be.

File Server option 3: FTPd

FTP for you, eh? Here we go. FreeBSD comes with an ftp group but not such a user by default. Let’s create it:

# pw useradd -n ftp -u 14 -c "FTP user" -d /var/ftp -g ftp -s /usr/sbin/nologin

It’s a convention that public data offered via anonymous FTP is placed in a “pub” directory. We’re going to honor that tradition and create the directory now:

# mkdir -p /var/ftp/pub
# chown ftp:ftp /var/ftp/pub

If you intend to use logging, create an empty initial log file:

# touch /var/log/ftpd

Now we need to enable the FTP service for inetd (the “l” flag enables a transfer log, “r” is for operation in read-only mode, “A” allows for anonymous access and “S” is for enabling the download log):

# echo "ftp     stream  tcp     nowait  root    /usr/libexec/ftpd       ftpd -l -r -A -S" >> /etc/inetd.conf

As we intend to run a service that allows local users to log in via FTP, we need to consider the security implications of this. In my case I have created the “kraileth” user and it has the power to become root via doas. While OpenSSH is configured to only accept key-based logins, FTP is not. The user also has a password set – which means that an attacker who suspects that the user might exist, can try brute-forcing my password.

If you’re the type of person who is re-using passwords for multiple things, take this scenario into consideration. Sure, this is an internal server and all. But I recommend to get into a “security first” mindset and just block the user from FTP access, anyway. To do so, we just need to add it to the /etc/ftpusers file:

# echo "kraileth" >> /etc/ftpusers

Now let’s restart the inetd service (as it’s already running for TFTP) and check it:

# service inetd restart
# sockstat -4l | grep 21
root     inetd      1464  7  tcp4   *:21                  *:*

Ready and serving!

Optional: Checking File Server service

Time for a final test. If you’re using a webserver, do this:

# echo "TestTest" > /usr/local/www/pxe/test
# fetch http://10.11.12.1/test
test                                             9  B 8450  Bps    00s
# cat test
TestTest
# rm test /usr/local/www/pxe/test

If you’re using FTP instead:

# echo "TestTest" > /var/ftp/pub/test
# fetch ftp://10.11.12.1/pub/test
test                                                     9  B 9792  Bps    00s
# cat test
TestTest
# rm test /var/ftp/pub/test

Everything’s fine here, so we can move on.

What’s next?

In part 3, we’re finally going to add data and configuration to boot multiple operating systems via PXE.

Multi-OS PXE-booting from FreeBSD 12: Introduction (pt. 1)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/multi-os_pxe-booting_from_fbsd_pt1.gmi

This is an introductory article; if you’re familiar with PXE you will want to skip the excursion but may be interested in the “Why”. The article ends with the post-installation setup of my test machine, turning it into a simple router so that the next article can start with the actual PXE-related setup.

Also see part 2 here and part 3 here as well as part 4 here.

Situation

These days I decided to reactivate an old laptop of mine. It has processor of an older generation (Ivy Bridge) but it’s an i7 – and it has 24 GB of RAM, making it a somewhat nice machine for certain things. Problem with it is: The USB does not work (I think that the controller was fried)!

So how do I get a current OS installed on there as I cannot use the CD emulator I’d normally use? Sure, I could always burn a CD, but I don’t feel like it (unnecessary waste). I could open it up, remove the drive, place it in another machine, install the OS and then put it back. Nah, not too much fun, either. How about doing a network install then?

When I thought of that I realized that I had some old notes from about a year ago somewhere on my workstation. I originally meant to blog about that topic but never actually wrote the post. So here’s how to do what the post’s title says – updated to FreeBSD 12.2 and Ubuntu 20.04!

While I have a clear favorite OS and very much appreciate what FreeBSD has to offer for me as the administrator of a fleet of servers, there really is no reason to turn a blind eye to other Unix-like systems. For one thing it totally makes sense to always pick the best candidate for a job (which might not in all cases be one OS). That and the fact that you cannot judge which one is best suited if you don’t have at least some level of familiarity with various options. But in addition to that my employer runs a heterogeneous environment, anyway – and while I’m mostly occupied with the BSD side of things, there’s simply way, way too many Linux machines around to even think about avoiding them altogether all the time.

Why PXE-boot Linux from FreeBSD?

After an update, the old means of installing Linux servers as we do it at work had stopped working reliably. I looked at it briefly but soon found that too many things weren’t set up like you’d do it today. Therefore I decided that it might make more sense to start fresh. And while at it – wouldn’t it make sense to try and combine the Linux and FreeBSD PXE servers on one machine? That would mean one less server to run after all.

The next installation due was for a customer who requested an Ubuntu server. As a matter of fact we are transitioning to use that distribution more often for our internal infrastructure, too (decision by management, certainly not my choice!). For that reason one weekend I ended up doing something that I hadn’t done in a while: installing a fresh Ubuntu 16.04 system on a test machine I. After doing this I could write a whole post about how bad the installer actually is (static IP addressing, anyone??), but I don’t want this to turn into a rant.

So let’s just mention one single complaint: Installing takes quite a long time (I never understood why Debian and derivatives install stuff that they remove again later during the “cleaning up” phase…). Before the installation was even finished, I admittedly already had mixed feelings about this new system. But then this was what happened on the first boot:

[ TIME ] Timed out waiting for device dev-disk-by\x2duuid-0ef05387\x2d50d9\x2d4cac\x2db96\x2d9808331328af.device.
[DEPEND] Dependency failed for /dev/disk/by-uuid/0ef05387-50d9-4cac-b796-9808331328af.
[DEPEND] Dependency failed for Swap.
[  *** ] A start job is running for udev Coldplug all Devices (3min 34s / no limit)

You’ve got to be kidding! A freshly installed system going sideways like that? Sorry Ubuntu that’s not the kind of stuff that I like wasting my time with! I don’t even care if it’s systemd’s fault or something else. The boss “preferably” wanted an Ubuntu system – I gave it a try and it failed me. Ah, screw it, let’s deploy something I know of that it actually works (same hardware BTW, before anybody wants to blame that for a problem that’s definitely Ubuntu’s fault)!

I had a freshly installed FreeBSD with static IP configuration (which you probably want to use for a DHCP / PXE boot server after all) in a fraction of the time that the Ubuntu installation took. And it goes without saying: System starts as one would expect.

Excursion: PXE – An Overview

While there have been earlier methods for making a machine boot over the network, PXE (Preboot eXecution Environment) is what you want to rely on if you need to do that today. It is a specification widely implemented (in fact just about everywhere) and chances are very low that you will run into problems with it. Have a look around in your computer’s EFI or BIOS, often PXE-booting is disabled (and if you want to use it just once to install an OS on the machine, I recommend that you disable it again afterwards). How to make a machine boot over the net depends on its EFI / BIOS. Figure out how to do it four your metal and give it a try.

On your average home environment not too much should happen. The PXE environment will probe the network for the information that it needs, receive none and eventually give up. But what information does it need and which components are involved?

Well, it needs to load an operating system “from the net” – which means from another machine. To be able to talk to other hosts on the network, it needs to become part of said net. For this it needs to know the network parameters and requires a usable, unique IP address for itself. It can get all of that from a DHCP (Dynamic Host Configuration Protocol) server. If configured correctly, the daemon will accept DHCP requests and provide the client with a suggested IP as well as information about the network (like the netmask, default router, nameservers on the net, etc). It can also tell the client from where it can load the NBP (Network Bootstrap Program).

The latter is a piece of software, usually a special bootloader. It will be downloaded via TFTP (Trivial File Transfer Protocol). Think of it as very similar to FTP but much, much more primitive. The NBP is executed when the transfer is completed. It will then download its configuration file and then act accordingly, most likely fetching an operating system kernel and additional files, then booting the kernel.

TFTP is what the PXE bootcode speaks and uses. But due to its very simplistic nature, TFTP is not well fit for larger file transfers. Therefore such files are usually loaded via other means once the kernel has booted and more options are available. FreeBSD likes to use NFS, while on Linux HTTP is often used to receive a filesystem or installation medium image.

So the following services are involved when setting up a PXE boot server:

  • DHCP server
  • TFTP server
  • Webserver / NFS service stack

Preparing a FreeBSD router

Today I’m redoing my previous work in my home lab with FreeBSD 12.2. The machine that I’m using is pretty old (32-bit only Atom CPU!) but it has 6 Ethernet ports and still works well for network tinkering. I got it for free some years ago, so there’s really no reason to complain.

When it comes to the installation, in this case I’m going with MBR + UFS which I normally wouldn’t do for a amd64 machine. It’s a standard 12.2 installation otherwise with my “kraileth” user (member of the wheel group) added during the installation.

First thing to do is copying my public SSH key onto the server and then SSHing into the machine:

% ssh-copy-id -i ~/.ssh/id_ed25519 kraileth@hel.local
% ssh kraileth@hel.local

Now I switch to the root user, disallow SSH password-based login and restart the SSH daemon:

% su -l
# sed -i "" 's/#ChallengeResponseAuthentication yes/ChallengeResponseAuthentication no/' /etc/ssh/sshd_config
# service sshd restart

I don’t need sendmail on the machine, so off it goes:

# sysrc sendmail_enable="NONE"

Next is bootstrapping the package manager, installing the unbound DNS server and – for convenience – doas (a simpler sudo replacement). Writing a one-line config for doas is fine for my use case:

# env ASSUME_ALWAYS_YES=TRUE pkg bootstrap
# pkg install -y doas unbound
# echo "permit :wheel" > /usr/local/etc/doas.conf

Then unbound needs to be configured. To do so, edit the config file like this:

# vi /usr/local/etc/unbound/unbound.conf

Delete all the content, put the following in there (adjust to your network, of course) and save:

server:
        verbosity: 1
        interface: 10.11.12.1
        access-control: 10.11.12.0/24 allow

Keep in mind that this is not a production configuration! You will have to do some research on the program if you want to put it into production. Do at the very least read the more than thousand lines of comments that you just deleted (you can find it in /usr/local/etc/unbound/unbound.conf.sample). The above config will make the daemon bind on the IP configured via the “interface” statement and allow DNS recursion for any host in the subnet defined via “access-control”. Let’s enable the daemon now (so it will start after rebooting the machine):

# sysrc unbound_enable="YES"

During installation I chose the first NIC (em0) to be configured via DHCP as offered by my home router. This machine will act as a DHCP server for another network, so I assign a static address to the respective interface (em5) that I have connected to a spare switch I had around. It will also act as a gateway between the two networks it is part of:

# sysrc ifconfig_em5="inet 10.11.12.1 netmask 255.255.255.0"
# sysrc gateway_enable="YES"

To actually provide Internet connectivity to hosts in the 10.11.12.0/24 subnet, we have to enable NAT (Network Address Translation). Otherwise any host that is contacted via an IP in that range will have no idea how to answer, even though our gateway forwarded the message to it. NATing is done via the firewall. FreeBSD offers three of them. Pf is my absolute favorite, but ipfw is ok, too. I wouldn’t recommend ipf – it’s old and there is really no reason to use it if you’re not already doing so and don’t want to switch.

First we’ll create a configuration file:

# vi /etc/pf.conf

Now put the following in there (again: Adjust to your network) and save:

ext_if="em0"
int_if="em5"
localnet = $int_if:network

scrub in all
nat on $ext_if from $localnet to any -> ($ext_if)

This defines commonly used three macros: One for the external and internal NIC and one for the local network. It then turns scrubbing on (you almost always want to do this and I suggest that you look up what it does). The final line does the magic of the actual NAT.

You will want to define a more complex firewall configuration if you plan to use this gateway in production. But this is enough for my (demonstration) purposes. I suggest that you follow along and when you made sure that PXE booting works, tighten up security – then test if everything still works. Firewalling is a topic of its own, and in fact not a small one, though. If you’re new to it it makes perfect sense to do some reading. There’s lots of free material on the net as well as a great “Book of Pf” by Peter Hansteen if you prefer.

So let’s enable Pf on startup:

# sysrc pf_enable="YES"
# sysrc pf_rules="/etc/pf.conf"

Reminder: If you want to do things right, you will probably want to enable pflog, too, for example. This post is meant to point you in the right direction and to get stuff to work quickly. Familiarizing yourself with the various components used is your own homework.

It’s always a good idea to update the system to the latest patchlevel:

# freebsd-update fetch install
# freebsd-version -kr
12.2-RELEASE-p1
12.2-RELEASE

Ok, looks like kernel-related updates were installed, so it’s time to reboot:

# shutdown -r now

That’s it for the preparation work. Once the server comes back up again it should function as a router. If you configure a host with a static address in the 10.11.12.0/24 network, it should be able to have DNS queries answered as well as reach the Internet.

What’s next?

In part 2 we’re going to complete the setup of the PXE boot server so that a client can boot into Ubuntu Linux. The rest will be in part 3.

CentOS killed by IBM – a chance to go new ways?

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2020/centos_killed.gmi

On December 8 2020 a Red Hat employee on the CentOS Governing Board announced that CentOS would continue only as CentOS Stream. For the classical CentOS 8 the curtain will fall at the end of 2021 already – instead of 2029 as communicated before! But thanks to “Stream” the brand will not simply go away but remain and add to the confusion. CentOS Stream is a rolling-release distribution taking a middle grounds between the cutting-edge development happening in Fedora and the stability of RHEL.

Many users feel betrayed by this action and companies who have deployed CentOS 8 in production trusting in the 10 years of support are facing hard times. Even worse for those companies who have a product based on CentOS (which is a pretty popular choice e.g. for Business Telephone Systems and many other things) or who offer a product targeted only at RHEL or CentOS. For those the new situation is nothing but a nightmare come true.

What do we do now?

IBM-controlled Red Hat obviously hopes that many users will now go and buy RHEL. While this is certainly technically an option I would not suggest going down that road. Throwing money at a company that just killed a community-driven distribution 8 years early? Sorry Red Hat but nope. And Oracle is never an option, either!

The reaction from the community has been overwhelmingly negative. Many people announced that they will migrate to Debian or Ubuntu LTS, some will consider SLES. Few said that they will consider FreeBSD now – being a happy FreeBSD user and advocate this is of course something I like to read. And I’d like to help people who want to go in that direction. A couple of weeks ago I announced that I’d write a free ebook called The Penguin’s Guide to Daemonland: An introduction to FreeBSD for Linux users. It is meant as a resource for somewhat experienced Linux users who are new to FreeBSD (get the very early draft here – if anybody is interested in this feedback is welcome).

But this post is not about BSD, because there simply are cases where you want (or need) a Linux system. And when it comes to stability, CentOS was simply a very good choice that’s very hard to replace with something else. Fortunately Rocky Linux was announced – an effort by the original founder of CentOS who wants to basically repeat what he once did. I wish the project good luck. However I’d also like to take the chance as an admin and hobby distro tinkerer to discuss what CentOS actually stood for and if we could even accomplish something better! Heresy? Not so much. There’s always room for improvement. Let’s at least talk about it.

Enterprise software

The name CentOS (which stood for Community Enterprise Operating System) basically states that it’s possible to provide an enterprise OS that is community-built. We all have an idea what a community is and while a lot could be written about how communities work and such I’d rather focus on the other term for now. What exactly is enterprise? It helps to make a distinction of the various grades of software. Here’s my take on several levels of how software can be graded:

  • Hobbyist
  • Semi-professional
  • Professional
  • Enterprise
  • Mission critical and above

Hobbyist software is something that is written by one person or a couple of people basically for fun. It may or may not work, collaboration may or may not be desired and it can vanish from the net any day when the mood strikes the decision maker. While this sounds pretty bad that is not necessarily the case. Using a nifty new window manager on your desktop is probably ok. If the project is cancelled tomorrow it just means that you won’t get any more bug fixes or new features and you can easily return to your previous WM. But you certainly don’t want to use such software in your product(s).

Semi-professional software is developed by one person or a team that is rather serious about the project and aiming for professionalism (but commonly falling short due to limitations of time and resources). Usually the software will at least have releases that follow a versioning scheme, source tarballs will not be re-rolled (re-using the same version number). There will be at least some tests and documentation. Patches as well as feedback and reporting issues are almost certainly welcome. The software will be properly licensed, come with things like change logs and release notes. If the project ends you won’t know because the repo on GitHub was deleted but because at least an announcement was made.

Professionally developed software does what the former paragraph talked about and more. It has a more complex structure with multiple people having commit rights so that a single person e.g. on holiday when a severe bug is found doesn’t mean nobody can fix things for days. The software has good test coverage and uses CI. There are several branches with at least the previous one still receiving bug fixes while the main development takes place in a newer branch. Useful documentation is available and there is a table with dates that show when support ends for which version of the software. There’s some form of support available. Such software is most often developed or at least sponsored by companies.

Enterprise products take the whole thing to another level. Enterprise software means that high-quality support options (most likely in various languages) are available. It means that the software is tested extensively and has a long life cycle (long-term support, LTS). There is probably a list of hardware on which the software is guaranteed to work well. And it’s usually not exactly cheap.

“Mission critical” software has very special requirements. For example it could be that it has to be written in Spark (a very strict Ada dialect) which means that formal verification is possible. Most of us don’t work in the medical or aerospace industries where lives and very expansive equipment may be at stake and fortunately we don’t have to give such hard guarantees. But without going deeper into the topic I wanted to at least mention that there’s something beyond enterprise.

Community enterprise?

Having a community-run project that falls into the professional category is quite an accomplishment even with some corporate backing. Enterprise-grade projects seem to be very much without reach of what a community is able to do. Under the right circumstances however it is doable. CentOS was such a project: They didn’t need to pay highly skilled professionals to patch the kernel or to backport fixes into applications and such. Thanks to the GPL Red Hat is forced to keep the source to the tools they ship open. The project can build upon the paid work of others and create a community-built distribution.

Whenever this is not the case the only option is probably to start as professional as possible and create something that becomes so useful to companies that they decide to fund the effort. Then you go enterprise. Other ways a project can go are aiming to become “Freemium” with a free core and a paid premium product or asking for donations from the community. Neither way is easy and even the most thoughtful planning cannot guarantee success as there’s quite a bit of luck required, too. Still, good planning is essential as it can certainly shift the odds in favor of success.

Another interesting problem is: How to create a community of both skilled and dedicated supporters of your project? Or really: Any community at all? How do you let people who might be interested know about your project in the first place? It requires a passionate person to start something, a person that can convince others that not only the idea is worthwhile but also that the goal is in fact achievable and thus worth the time and effort that needs to be invested.

A stable Linux OS base

As mentioned above, I’m a FreeBSD user. One of the things that I’ve really come to appreciate on *BSD and miss on Linux is the concept of a base system (Gentoo being FreeBSD-inspired kind of has “system” and “world”, though). It’s the components of the actual operating system (kernel and basic userland) developed together in one repository and shaped in a way to be a perfect match. Better yet, it allows for a clean separation of the OS and software from third party packages whereas on Linux the OS components are simply packages, too. On FreeBSD third party software has its own location in the filesystem: The operating system configuration is in /etc, the binaries are in /{s}bin and /usr/{s}bin, the libraries in /lib and /usr/lib, but anything installed from a package lives in /usr/local. There’s /usr/local/etc, /usr/local/bin, /usr/local/lib and so on.

Coming from a Linux background this is a bit strange initially but soon feels completely natural. And this separation brings benefits with it like a much, much more solid upgrade process! I maintain servers that were setup as FreeBSD 5.x in the mid 2000’s and still happily serve their purpose today with version 12.x. OS (and hardware) has been upgraded multiple times but the systems were never reinstalled. This is an important aspect of an enterprise OS if you ask me. I wonder if we could achieve it on Linux, too?

Doing things the BSD way would mean to download the source packages for the kernel, glibc and all the other libraries and tools that would form a rather minimal OS, extract them and put the code in a common repository. Then a Makefile would be written and added that can build all the OS components in order with a single command line (e.g. “make buildworld buildkernel”). Things like ALFS (Automated Linux From Scratch) already exist in the Linux world, so building a complete Linux system isn’t something completely new or revolutionary.

As vulnerabilities are found fixes are committed to the source repo. It can then be cloned and re-built. Ideally we’d have our own tool “os-update” which will do differential updates to bring the OS to the newest patch level. My suggestion would be to combine the components of a RHEL release – e.g. kernel 4.18, glibc 2.28, etc. following RHEL 8. So more or less like CentOS but more minimal and focused on the base system. That system would NOT include any package manager! This is due it being intended as a building block for a complete distribution which adds a package manager (be it rpm/dnf, dpkg/apt, pacman or something else) on top and consumes a stable OS with a known set of components and versions, allowing the distributors to focus on a good selection of packages (and giving them the freedom to select the means of package management).

Just to give it a working title, I’m going to call this BaSyL (Base System Linux). For the OS version I would prefer something like the year of the corresponding RHEL release, eg. BaSyL 2019-p0 for RHEL 8. The patch level increases whenever security updates need to be applied.

Enterprise packages

The other part of creating an enterprise-like distribution is the packages. Let’s call it the SUP-R (Stable Unix-like Package Repo) for now. So you’ve installed BaSyL 2019 and need packages. There’s the SUP-R 2019 repo that you can use: It contains the software that RHEL ships. Ideally a team forms that will create and maintain a SUP-R 2024 repo after five years allowing to optionally switch to newer package versions. The special thing here would be that this SUP-R 2024 package set would be available for both BaSyL 2019 and BaSyL 2025 (provided that’s when the next version is released). That way BaSyL 2019 users that already made the switch to SUP-R 2024 can say on that package set when updating the OS and don’t have to do both at the same time! A package repo change requires to consult the update documentation for your programs: E.g. Apache HTTPd, PostgreSQL database, etc. Most likely there are migration steps like changing configuration and such.

Maintaining LTS package sets is not an easy task, though. However there are great tools available to us today which makes this somewhat easier. Still it would require either a really enthusiastic community or some corporate backing. Especially the task of selecting software versions that work well together is a pretty big thing. It would probably make sense to keep the package count low to start with (quality over quantity) and think of test cases that we can build to ensure common workloads can be simulated beforehand.

In addition to experienced admins for software evaluation, package maintainers and programmers for some patch backporting, also people writing docs (both in general and for version migration) would be needed. Yes, this thing is potentially huge – much, much more involved than doing BaSyL alone.

Conclusion

I’d love to see a continuation of a community-built enterprise Linux distro worth the title. But at the same time getting to know BSD in addition to Linux made me think a little different about certain things. De-coupling the OS and the packaging efforts could open up interesting possibilities. At the same time it could even help both projects succeed: Other operating systems might also like (optional) enterprise package sets. And since they’d be installed into a different prefix (e.g. /usr/supr) they would not even conflict with native packages in e.g. Debian or any other glibc-based distribution.

If a portable means of packaging was chosen it would potentially also be interesting to the BSDs and Open-Solaris derivatives. And the more potential consumers the larger the group of people who could have the motivation to make something like this come true.

This article is meant to be giving food for thought. Interested in talking about going new ways? Please share your thoughts!

Retreat… and reboot! Is there life outside of the Web?

Dear readers,

the time has come for some fundamental re-organization of my EerieLinux blog. The impact of this year’s pandemic as well as family matters (a new family member that demands quite a bit of care!) have forced me to post-pone many of my posts that I usually published at least monthly. Right when I was getting back on track and stated that I intended to publish the missed articles (I’ve written them after all but didn’t find the time for final editing) something else happened.

No, to be honest, it has been happening for quite a while. I didn’t like it but had to cope with it as there was nothing I could do about that, anyway. What am I talking about? Well, the sad fact that I would call the “degradation of the Web”.

EerieLinux has always been about some experiments that I did and wanted to share in case anybody was interested. Now I’m in for a new experiment, a somewhat radical one (and what could be considered radical after going Linux-only and later FreeBSD-only for all of my machines?). I would appreciate feedback on the path that I choose to give a try. What do you think about it? Are you interested in that, too? Would you continue to read some of my material or does this mean that our ways will part? But let’s detail first what has made the time ripe for what is less of an elitist move then an act of mere desperation.

What’s wrong with the Web?

So, what has happened? And what’s the gripe that I have with the Web? Well… It’s actually not just the Web (read: WWW), it’s more things coming together. There seems to be a general direction that the IT world is heading which I don’t approve of. This blog is powered by WordPress for the simple reason that when I started in 2012 I had some material that I wanted to share with the world and going with a free plan on a blog hosting platform seemed like a nice way to get started quickly. And it was. Mind the past tense here.

WordPress evolved – which is a good thing in general. It evolved in a way that I loathe, however. There have been all these “new” and “easier” modes for XYZ which were admittedly more shiny and more “modern” than before. But they broke my workflow time and time again – and unfortunately not for the better. I actually like to re-evaluate my workflows, like to challenge my habits. When I was still using Ubuntu Linux I didn’t only give Canonical’s Unity DE a chance, I actually tried to use it before putting it aside because in the end it hindered me more than it helped. I forced myself to use the (then) new graphical TrueOS for half a year and suffered through incompleteness and breakage. I have a fairly high tolerance for things that are… sub-optimal. I’m also rather patient, I think, and I’m surely not a rage-quit type of person. But inevitably there’s this point where I have to admit: “Ok, that’s it, that’s simply too much”.

The same thing happened for WordPress, when they came up with various iterations of “new editors” – which is why I opted to go back to the “old editor” for writing and editing my blog posts. As I returned to finish the next old post however, I found out that the old editor was gone. They had “finally” removed it. Alright, so I had to try and get accustomed to the new one again. Perhaps it had matured since I tried it last? It had. But it proved to be even more annoying for me than before! I don’t know what the designers are thinking – certainly not what I do. For me it is a pain to work with the new tool: Things that worked perfectly fine before are gone and the new options are so much inferior to what we had before… And this is not even the worst thing that made me give up when I was more than half through with polishing the next post.

Technically the new editors work well for small posts. I seem to have a bit more to say than the average blogger, though; and here’s a very frustrating issue: When editing longer posts, the editor gets unresponsive and laggy. Yuck! Trading a working one for a “new” one that does less but eats up so much more resources? Are you freaking kidding me? Oh, right, that’s the way things go, I forgot for a second. There’s something within my very self that refuses to accept the lack of sanity around me, sorry. I should know better by now.

And even though WordPress makes up a quite large portion of today’s Web, that’s just the tip of the iceberg. All this JS nonsense, megs and megs of useless graphics, animations, tracking and spying cookies, etc. pp. make the Web such a obnoxious place! And don’t even get me started on the situation with web browsers…

What’s wrong with hardware?

After my little rant I can hear people say: “See, pal… If the WordPress editor makes your system struggle enough that you complain on the net, you reaaally want to get a new PC! There’s machines with more than 4 gigs of memory today, you know.”

The irony here is: I did get a newer laptop – and went back to primarily use the old one! Why? Because the new one is bugging me just too much whereas the older model works well for me. The newer follows that stupid trend of being extremely thin. “Thanks” to that design no DVD drive fits in there. While this is incredibly dumb, I can live with it. Carrying an external drive with me is not so much of a problem, but still the fact that the machine is missing the drive annoys me. The bigger problem: The battery of the old model can easily be removed and I can carry a second one with me e.g. on longer train rides. With the new one I’d need to open the case to access it! WTF HP? And finally: Some idiot decided that the “cool” flat design was so important that it’s thinner than the jack for an Ethernet connector… Yes, really: There’s some kind of… hatch that I need to pull down to make the opening big enough to insert the plug! If I remove the plug the hatch snaps back up. It’s not hard to see that this brain-dead construction is prone to defects. And really, it didn’t take too long until it stopped working reliably and I’m randomly losing connectivity every other day… Sorry, this is garbage. Plain as that.

I have another older laptop – that one has 24 GB of RAM and two hard drives. Nice, eh? Not so much because it also has some of those gross EFI quirks. When one of the drives has a GPT partition scheme applied to it, that drive won’t even be detected by the POST anymore! So on that machine I’m stuck with MBR which is not so much fun if you’re using ZFS and want to encrypt your pool with GELI – it can be done, but the workarounds have the side-effects of not working with Boot Environments! True garbage again.

“Come on, why don’t you simply get a different model then?” Yeah, right. As I said, I type a lot, so I have some demands concerning the keyboard. Unfortunately some manufacturers seem to take pride in coming up with new ways of f*cking up the keyboard layout. There are models that simply dropped the CAPS LOCK key, because nobody uses that, right? Wrong! There are people who remap their keys, and being a NEO² typist (a keymap that allows for 6 layers) I need that key as an additional modifier! Take it away and that keyboard is utterly useless to me. But don’t current models like the X1 Carbon still have CAPS LOCK? They do, but there’s another oddity: They swap the CTRL and FN keys! Having CTRL as the leftmost key on the keyboard that I use as my daily driver and switching around in my head at 3 AM when there’s an alarm during my on-call duty is something I fail to do properly. I used a Thinkpad for about a month for on-call and it drove me completely mad. Sorry Lenovo. Your laptops may be excellent otherwise, but they are not for me.

Oh, and it’s not so much of a secret that the situation with the x86 and x86_64 architectures is abysmal (need I say more than Meltdown / Spectre?). Modern hardware design is fundamentally broken and I cannot say that I completely trust the fixes and what unknown side-effects they might bring with them either. But that’s not even the point. We’re stuck with market-leading technology that has been criticized as crappy right from the start. It has come a long way since then, but it’s not a great technology in any regard. I’ve been playing with my Pinebook and it might have the potential to become something different. Maybe the Open Source PowerPC notebook initiative will succeed. Probably we’ll have something RISC-V-based in the long run. But right now we have to make use of what is available to us.

Can’t the Web be saved?

I’m a somewhat optimistic person, but in this regard not so much. While I have thought about just self-hosting my blog and using a nice static-site generator or something, that would only solve my issues regarding the blog. A huge portion of the Web will still be a terrible place. Browsers and software will still suck hard. Especially in terms of browsers I’ve found even the suckless offering far from being a good one (I like their work and agree with most of the suckless principles, but in my conclusion it’s not enough to really get me out of this whole mess).

Sure, I can install e.g. the Dillo browser. But its development has been extremely slow for years and the subset of HTML that it supports cannot render most of today’s websites properly. And going with just the pages of people who deliberately keep their sites simple? I like the idea, but there’s the problem that no real community (that I know of) exists which agrees on a single standard of which HTML features are ok and which ones should be avoided.

I’m old enough to remember the browser wars well enough. I also confess being guilty of having used “marquee” on my homepage for a while back in the day. Maybe all of the gibberish we’re facing today is a late punishment for sins committed when we were younger? I don’t know.

So – are we doomed? Should we just pull the plug, go to real libraries again, buy books when we need information and spend our free time in the garden? That thought is tempting only for a moment. I love tech. I love the net and the possibilities it brings with it. Plus I’m kind of a liberal today who believes that mankind will eventually make something good from it all. We’re on this planet to play, to learn and to grow. Some insights can only be gained the hard way. It’s ok that commercial interests ruined the Web for people like me. There are other people who have (monetary or idealistic) interests in the Web being like it is. That’s fine. And in fact: Today’s Web “works” for a lot of people who either have no clue why it’s problematic in the first place or who simply don’t care. Which is a legitimate stance. The tragedy is simply that the rest of us has kind of lost home for a while and we haven’t been able to find a new one, yet.

The “Big Internet” vs. the “Small Internet”

While the Web makes up for a very large and definitely the most visible part of the Internet, there are niches, lesser known parts of it which are certainly not less interesting. Some people speak of the Web as the “Big internet” and call the various special parts the “Small Internet”.

One such niche is the so-called Gopherspace. Gopher is a simple communication protocol for documents which predates the Web. Not just by today’s standards it’s primitive – but it works. And while the major browsers either never supported it or removed support long ago (e.g. Firefox dropped Gopher support for release 4.0), it’s still kept alive by enthusiasts. Some sources even suggest that it has seen moderate growth over the last years with more people fed up with the Web trying out something different. If you’re interested, start here with the web proxy or do some research on your own and install a Gopher client.

I’ve dug into various Gopher holes and my experience with it is a mixed bag. On the one hand it’s really cool to see people putting in the effort of creating a place that’s worthwhile to visit. I also liked the experience in general: It’s not bells and whistles everywhere (because that’s in fact impossible due to the protocols limitations) but rather a focus on the actual content. People have found ways to access git repositories over Gopher and there’s a community called Bitreich (via Gopher) that declared “suckless failed” and propose an even more radical approach. While some of it is probably parody, it’s an interesting one.

On the other hand some of the restrictions of the protocol make no sense at all today, like having a hard-coded limit of 80 characters per line. And there are more shortcomings. When Gopher was conceived, nobody thought of internationalization and indicating the charset that a document uses. Also Gopher is all plain-text and does not provide any means to help protect your privacy – which does not exactly make it overly appealing to me.

But there’s something else, an even smaller – but newer – niche, called Project Gemini. It is a newly designed protocol (2019!) that wants to provide kind of a middle ground between Gopher and the Web while clearly leaning towards the former. It remedies the privacy issues by making TLS mandatory and deliberately not supporting “user agents” or other ways to collect data about the user not required to deliver the content. And it wants to simply be an additional option for people to use not to replace either Gopher or the Web.

While I kind of like Gopher, I really admire Gemini. Having the balls to even try to actually do something like this in our time is simply (no pun intended) astonishing. In my opinion it’s high time to re-invent the wheel. Gemini is certainly not a panacea to all the problems that we have today. But it’s a much welcome (for me at last) chance to start fresh without that huge, huge bag of problems that we’re carrying on with the Web but with the experience that we gained from using it for several decades now.

What’s next?

My experiment is to move Eerielinux to Geminispace and post new content there as I find the time. The plan is to keep this WordPress blog and to create new posts here that simply reference the real content, providing a convenient link using a Web proxy. That should help keeping things accessible for people who prefer to stick with their known browsers instead of getting another one for such a tiny little world as Geminispace is right now.

And who knows? Perhaps something nice comes from Gemini in the long run. I’m looking forward to find out. And if you care — see you in Geminispace for some more articles on *BSD, computers and tech in general!

Should you abandon Linux and switch to *BSD?

The popularity of Linux skyrockets these days: More and more companies adapt it and even the just-above-average who doesn’t accept the imposition that is called “Windows 10” is often open to try it out. However at the same time, the popularity of said System seems to be fading among some of the more technical people, operating system enthusiasts and even followers of the FLOSS ideology.

Just recently an article called Why you should migrate everything from Linux to BSD was published and it has caught some attention and even replies like this one (it makes false claims like ZFS not being available on NetBSD and such, though.)

There have always been some posts like this, this is nothing too special. What I think is new, however, is the frequency that you can find discussions like this. Also the general tone has changed. Just a couple of years ago most Linux users wouldn’t even have bothered to comment such a thing. Today they seem to be much more open to learning about alternatives or in fact looking for something better than what they have. So what’s wrong with Linux?

It’s not about Systemd (alone)…

There are a lot of perfectly valid technical reasons to not want to use Systemd on your Linux system. But none of those could ever be an excuse for the hatred that this project has attracted. However there is a pretty simple explanation for that phenomenon: It’s how somebody is acting all high and mighty, simply dismissing valid critics and being a great example for a person with an arrogant attitude.

We have a lot of… let’s say… difficult persons it tech. Sometimes you think they have no manners, are being jerks or greatly overestimate their knowledge in certain fields. That’s ok and in fact they often are somewhat brilliant in a certain area. Most of us have learned to live with that.

And then there are people who think that they can dictate the one way to go. Well, there are project leaders who actually can do that due to being widely respected. But sometimes it’s different. Now when somebody wants to take something away from you (or he really doesn’t but you get the impression), you are likely to stand upon your defense.

Now when all of that unites in one person, you have the perfect boogeyman. Then all the technical aspects lose in weight and the feeling takes over. Which is not to say that feeling is not an important factor: If you don’t feel comfortable with Linux anymore, it might be time to move on.

The GNOME factor

The GNOME desktop is well known among *nix desktop users. It suits the needs some people but not others. That’s fine and there would not be any problem at all. However GNOME has a certain reputation which is not that nice… Why? Not because they didn’t accept some feature requests. Also not because are being ignoramuses when it comes to systems that are not so mainstream. No. They are notorious for cutting features that have already existed! This is what makes a lot of people mad.

I was a GNOME user myself and remember pretty well how I liked the file manager. To my dismay they removed so many features which I needed that the application became useless for me. I didn’t want to go looking for something else, but I was eventually forced to as the situation became unbearable. As I’m more of a calm guy, I didn’t go off at insult anybody, but other people did. And things got worse…

Today I’m a former GNOME user. This is *nix and not Windows. Nobody can force tiles upon us against our will. Yes, some projects think they know better than their users and that leads to the latter becoming upset. But as long as there are alternatives, we can move on.

Clumsy leaders

Recently Linus Torvalds spoke out against ZFS. Being the Linux “inventor” he has earned a lot of respect among the *nix community. I also used to hold him in high esteem despite his often ignoble behavior. However over the course of the last few years, I’ve lost a lot of respect for him.

The ZFS statement was the last coffin nail for me. He says that he always found ZFS more of a “buzzword” than anything else and that “benchmarks” didn’t make it look that good anyway. This is so far off the shot that I’m ashamed to ever have considered him a technical genius. He obviously does have no idea at all what problem ZFS actually solves! Speed benchmarks are all nice and well, but they are not why people want ZFS! And of course it’s far from being a buzzword – if you have valuable data today, you almost certainly want to bet on ZFS.

But not only is he ok with judging what he has not even bothered to take a look at from closer that several hundred meters – he’s also making completely stupid claims that make him look like a terribly ridiculous figure. According to Mr. Torvalds, ZFS had “no real maintenance behind it either any more”.

Ouch! He hasn’t even heard about OpenZFS, I’d guess. If you’re not in a closed-Solaris environment, this is what people are referring to when they say “ZFS”. Nobody outside the small, isolated isle of Oracle has any interest in ClosedZFS. Yes, Oracle laid off most of their Solaris staff and nobody knows if there is any noteworthy future for that OS. But not too long Solaris 11.4 was released – so even if Linus referred to the situation at Oracle, he’s not exactly right. In the case of OpenZFS, however, he could not be more wrong.

The OpenZFS is as alive as it could be. There are regular leadership meetings, many new features are being developed – and just recently a common repository for both Linux and FreeBSD was created, with other operating systems expected to join in! This is a moment of glory for collaboration in Open Source, but Linus didn’t hear a thing – or did he not want to hear anything? The fact that the second-in-command, GKH, has attacked ZFS about a year ago in a pretty questionable way, too, does not bode well.

Why do people leave Linux?

There have always been compelling technical reasons why you would choose *BSD over Linux (e.g. the complete operating system approach, much more consistent design, etc.). But I’d say that lately the the feeling part of it became much more important.

I left Linux because I was so sick and tired of the stupid fights between the hardcore fans of one distro or another and the unbearable arrogance of many. Yes, I also had the feeling that Linux was heading down the wrong way, too. I simply was no longer really happy with it and ready to try something new. There was a learning curve for sure, but the FreeBSD community is extremely friendly and while there are of course also people getting into disputes, I got the feeling that I described as “BSD is for grown-ups”. Not saying that there aren’t any really bright people in the Linux community, but on average I feel that the BSD users are more technical.

Others have stated similar reasons. The primary developer of ClonOS (that strives to be for FreeBSD what Proxmox is for Linux) wrote this:

According to the authors of the project, Linux is no longer a member of the common people, it is fully controlled by big commercial organization. while FreeBSD is developed mostly by enthusiasts. Today, Linux – it is a commercial machine for making money – is that it was Microsoft Windows in 90 years. While many Linux users have struggled against the Windows monopoly (CBSD author of one of them).

Yes, FreeBSD very far behind in their characteristics in comparing to Linux. Just look at the abundance of such powerfull decisions as the OpenVZ, Docker, Rancher, Kubernetis, LXD, Ceph, GlusterFS, OpenNebula, OpenStack, Proxmox, ISPPanel and a dozen others. All this is created by commercial companies for Linux and this is done very well. However, Linux is oversaturated with similar solutions. Therefore, it’s much more interesting to create it on FreeBSD, where nothing like that exists. This is an excellent challenge to improve and fix in FreeBSD.

We all love independence and freedom and FreeBSD today – an independent and free operating system, which is in the hands of ordinary people.

They are not alone. Even convinced followers of the FSF ideas have come to the conclusion that Linux may not be the right platform for them anymore. The people behind Hyperbola GNU/Linux have announced this:

Due to the Linux kernel rapidly proceeding down an unstable path, we are planning on implementing a completely new OS derived from several BSD implementations. This was not an easy decision to make, but we wish to use our time and resources to create a viable alternative to the current operating system trends which are actively seeking to undermine user choice and freedom.

This will not be a “distro”, but a hard fork of the OpenBSD kernel and userspace including new code written under GPLv3 and LGPLv3 to replace GPL-incompatible parts and non-free ones.

Reasons for this include:

Linux kernel forcing adaption of DRM, including HDCP.
Linux kernel proposed usage of Rust (which contains freedom flaws and a centralized code repository that is more prone to cyber attack and generally requires internet access to use.)
Linux kernel being written without security and in mind. (KSPP is basically a dead project and Grsec is no longer free software)
Many GNU userspace and core utils are all forcing adaption of features without build time options to disable them. E.g. (PulseAudio / SystemD / Rust / Java as forced dependencies)

As such, we will continue to support the Milky Way branch until 2022 when our legacy Linux-libre kernel reaches End of Life.

Will *BSD be a better OS for you?

So the big question is: If you are a Linux user, should you make the switch, too? I won’t unconditionally say yes. It really depends.

Are you happy with the overall situation in Linux? In that case there’s no need to migrate anything over. However you might still want to give a BSD of your choice a try. Perhaps you find something that you like even better? If you spend a bit of time exploring a BSD, you will find that several problems can be solved in other ways than those you are familiar with. And that will likely make you a better Linux admin, even if you decide to stick with it. Or maybe you’ll want to use the best tool for the job which could sometimes be Linux and sometimes a BSD. Getting to know a somewhat similar but also at times quite different *nix system will enable you to make an informed choice.

Not happy with Linux anymore in the recent years? Try out a BSD. If you need help to decide which one might be for you, I’ve written an article about that topic, too. Do a bit of reading, then install that BSD in a VM and explore. If you go with FreeBSD, make sure you take a look at the handbook (probably also available in your language) as that is a great source of information and one thing that sets FreeBSD apart from almost all Linux distros.

If you find that you like what you found, make a list of your requirements and find out if your BSD would indeed fulfill your needs. If it doesn’t, consider alternatives. Once the path is clear, I recommend to take a look at the community, too. For example there’s the weekly BSDNow! podcast that’s very informative. A lot of people have already written in, confessing that they are still Linux users only, but the topics of the show got them still hooked.

Do not rush things. Did you start with Linux or have you migrated e.g. from Windows? If you did come from a different OS, remember that there have been frustrating moments when you were all new to Linux and had certain misconceptions. You will be going through that again, but looking at the final outcome it will likely be a pretty rewarding journey.

Also don’t be shy and ask others if you don’t have the time or will to figure out everything yourself. The BSD people are usually pretty approachable and helpful. Feel free to ask me questions here, I might be able to give some answers.

It has been a couple of years now since I replaced the last machine that ran Linux at home. Would I choose to make the switch after all the experience that I gained since then? Oh yes! Anytime.

A glimpse into 2020

When you read this, the old year will be over (well, depending on the time zone you live in). If we’re lucky, this might be the year to get our hands on the first affordable RISC-V hardware that can actually run a Unix-like operating system. It should definitely be the year to get interesting devices like the ARM64-based PinePhone. And it also means that Python 2 is finally dead.

Speaking about that: For me 2019 has been a pretty busy year. On this blog I wrote about quite some different topics, among them my first attempt at writing something programming-related as I tried to teach myself a little bit of Python. If I had to name an overall theme, I’d say that the past year was the year of hardware architectures. I didn’t plan this, but that’s what’s happened. But I don’t actually want to look back in this post. On the contrary! But speaking of dead things also kind of fits into the next topic (more than I like…)!

FreeBSD on SPARC64

The next post that I plan to write will be about FreeBSD on the SPARC64 architecture. What I did not know when I decided on that is that it is more or less a doomed architecture when it comes to FreeBSD. SPARC64 is in grave danger – people expect support for it to be dropped before FreeBSD 13.0 is released!

The reason is that it is one of the architectures that still need the old GCC 4.2 (yes, from 2007!) toolchain – and that old cruft finally has to go. And while everybody agrees that this is a completely sensible thing to do, SPARC64 doesn’t seem to have as many friends among the FreeBSD developers to make the transition to something newer. A few people are trying to get something done (I’m also tinkering and trying to help), but it’s far from a save bet that it’ll succeed.

IMHO it would be a real shame to see FreeBSD on SPARC64 die. If it does survive, I’ll definitely try to help with QA. Can you help? If so: Please do! I’ll give more details on what the current status is and where the problems are in my January post.

FreeBSD on ARM64

I plan on writing another post about the current status on FreeBSD on ARM64. The topic of making it a Tier 1 architecture has recently been brought up again and I’d like to join the discussion about that rather sooner than later. If it wasn’t for the very unpleasant situation with SPARC64, this would actually be my next post.

Progress has been made on the networking issues on the Cavium ThunderX servers and I’ll also take a look again at the PineBook. Most likely I’ll also buy a PinePhone and/or one of their Tablets. If I do, you will find a review here.

Orchestration and Configuration Management with SaltStack

I wanted to write about this topic for a while, but now I’ve at last started to set aside the hardware that I need for the project. Several years ago (gosh…) I wrote a little comparison of Puppet, Chef, Salt and Ansible. After a general introduction I might update and publish this as well.

But that will only be the start of a series of posts introducing Salt. There will be a slight focus on FreeBSD, but in general it will show off how to work with various operating systems and distributions. We’ll start with Salt-SSH using Remote Execution Modules, talk about targeting and get to know grains. Then we’ll progress to the state system, pillar data and so on before switching to the master-minion model.

I’m looking forward to this one. If anybody has any ideas – just tell me, I’m open to suggestions on what to cover.

HardenedBSD

While I ran this derivative of FreeBSD for a couple of weeks on spare hardware (until I needed that for something else), I just didn’t find the time to write about my experience with it, yet. I liked it, though, and plan on re-visiting the OS. And when I do, you’ll read about it here for sure.

The project is currently being re-structured. So I’ll wait with this topic for a while longer. Might happen in the second half of the year (time flies by much too fast, anyway).

E-Mail

For years now I’ve been putting this one off. In fact I’ve started digging a bit into the topic twice. Two times I got distracted with other topics. Maybe third time is the charm?

To help making this more likely, I registered a domain (with a pretty bad pun in German) in December. Maybe now that I pay money, I’ll actually live up to the plan to “do moar with mail”. I’ll be using BSD technology where possible. So expect that the mailing stuff will involve OpenSMTPd.

illumos

This one is really a time issue. I’m still very much interested in the heritage of OpenSolaris and would like to do some more things with it. However I have no idea when I’ll find the time to do a dedicated illumos project. But there will definitely be some illumos involved with one of the other topics. You guess which one! 😉

Ravenports

The Ravenports project is still as fascinating to me as it was when I discovered it. I really wish I could dedicate more time to porting and helping to bring things forward (there’s still quite a lot of ports missing that I’d like to have and use).

While things are going well in general and ports are being updated really, really fast most of the time, big changes are rare right now. But big changes are what it makes sense to write about. And while there are some noteworthy things that I can think of, I’m still waiting for something else to land. Once that happens I will dedicate another post to Raven.

Linux vs. FreeBSD

I recently needed to setup a new Linux machine for a customer. Usually my co-workers do that, since I volunteered to take care of our BSD machines. That installation left me totally puzzled. Has the Debian installer become worse – or does my memory fail me and it has always been so bad (and I didn’t notice when I was into Linux only)?

Since then I thought about a few things. The conclusion is that I really love FreeBSD. It’s not perfect (well, nothing is), but there are so many areas where it’s much, much more comfortable to work with (can you say iptables or mdadm? Yuck!). And there is a lot more beauty and even technical genius if you take a closer look and compare things.

Yes, Linux in much more advanced in many areas. But that’s not much of a surprise given how much more manpower goes into that system. But it is a little miracle how the BSDs with their much lower manpower continue to deliver excellent operating systems on par with or even superior to Linux when it comes to sanity of use. Thank you, *BSD!

Happy new year!

So that’s what I have on my mind right now (I’m not out of ideas, but these are the topics that are on the top of my “would like to write about” list currently). Which of these topics will I be able to deliver and which will I miss? Time will tell. Feel free to comment and tell me what interests you the most.

Happy new year to all of my readers!

Writing a daemon using FreeBSD and Python pt.3

Part 1 of this series covered Python fundamentals, signal handling and logging. We wrote an init script as well as a program that can be daemonized by daemon(8).

In the previous part we modified the program as well as the init script so that it can daemonize itself using the Python daemon module. I also covered a few topics that people totally new to programming (or Python) might want to know to better understand what’s happening.

Part 3 is about exploring a simple means of IPC (inter-program communication) by using named pipes.

Creating a named pipe

What is a named pipe – also known as a fifo (first in, first out)? It is a way of connecting two processes together, where one can sequentially send data and the other receives it in exactly the same order. It’s basically what us Unix lovers use for our command lines all the time when we pipe the input of one program into another. E.g.:

ls | wc -l

In this case the output of ls is piped to wc which will then print the amount of lines to stdout (which could be used as input for another program with another pipe). This kind of pipe between two programs is usually short lived. When the first program is done sending output and the second one has received all the data, the pipe goes away with the two processes. It also only exists between the two processes involved.

A named pipe in contrast is something a bit more permanent and more flexible. It has a representation in the filesystem (which is why it’s a named pipe). One program creates a named pipe (usually in /var/run) and attaches to the receiving end of the pipe. Another process can then attach to the sending end and start putting data into it which will then be received by the former. Named pipes have their own character (p) showing that a file is of type named pipe, looking like this when you ls -l:

prw-rw-r--

Here’s what the next version of the code looks like:

#!/usr/local/bin/python3.6
 
 # Imports #
import daemon, daemon.pidfile, logging, os, signal, time
 
 # Globals #
IN_PIPE = '/var/run/bd_in.pipe'
 
 # Fuctions #
def handler_sigterm(signum, frame):
    logging.debug("Caught SIGTERM! Cleaning up...")
    if os.path.exists(IN_PIPE):
        try:
            os.unlink(IN_PIPE)
        except:
            raise
    logging.info("All done, terminating now.")
    exit(0)

def start_logging():
    try:
        logging.basicConfig(filename='/var/log/bdaemon.log', format='%(levelname)s:%(message)s', level=logging.DEBUG)
    except:
        print("Error: Could not create log file! Exiting...")
        exit(1)

def assert_no_pipe_exists():
    if os.path.exists(IN_PIPE):
        logging.critical("Cannot start: Pipe file \"" + IN_PIPE + "\" already exists!")
        exit(1)

def make_pipe():
    try:
        os.mkfifo(IN_PIPE)
    except:
        logging.critical("Cannot start: Creating pipe file \"" + IN_PIPE + "\" failed!")
        exit(1)
    logging.debug("Created pipe \"" + IN_PIPE)

 # Main #
with daemon.DaemonContext(pidfile=daemon.pidfile.TimeoutPIDLockFile("/var/run/bdaemon.pid"), umask=0o002):
    signal.signal(signal.SIGTERM, handler_sigterm)
    start_logging()
    assert_no_pipe_exists()
    make_pipe()

    logging.info("Baby Daemon started up and ready!")
    while True:
        time.sleep(1)

We’re using a new import here: os. It gives the programmer access to various OS-dependent functions (like pipes which are not existent on Windows for example). I’ve also added a global definition for the location of the named pipe.

The next thing that you’ll notice is that the signal handler function got some new code. Before the daemon terminates it tries to clean up. If the named pipe exists the program will attempt to delete it. I’m not handling what could possibly go wrong here as this is just an example. That’s why in this case I just re-raise the exception and let the program error out.

Then we have a new “start_logging()” function that I put the logging stuff into to unclutter main. Except for that changed structure, there’s really nothing new here.

The next new function, “assert_no_pipe_exists()” should be fairly easy to read: It checks if a file by the name it wants to use is already present in the filesystem (be it as a leftover from an unclean exit or by chance from some other program). If it is found, the daemon aborts because it cannot really continue. If the filename is not taken, however, “make_pipe()” will attempt to create the named pipe.

The other thing that I did was moving the main part back from being a function directly to the program. And since we’re doing small incremental steps, that’s it for today’s step 1. Fire up the daemon using the init script and you should see that the named pipe was created in /var/run. Stop the process and the pipe should be gone.

Using the named pipe

Creating and removing the named pipe is a good first step, but now let’s use it! To do so we must first modify the daemon to attach to the receiving end of the pipe:

#!/usr/local/bin/python3.6
 
 # Imports #
import daemon, daemon.pidfile, errno, logging, os, signal, time
 
 # Globals #
IN_PIPE = '/var/run/bd_in.pipe'
 
 # Fuctions #
def handler_sigterm(signum, frame):
    try:
        close(inpipe)
    except:
        pass

    logging.debug("Caught SIGTERM! Cleaning up...")
    if os.path.exists(IN_PIPE):
        try:
            os.unlink(IN_PIPE)
        except:
            raise
    logging.info("All done, terminating now.")
    exit(0)

def start_logging():
    try:
        logging.basicConfig(filename='/var/log/bdaemon.log', format='%(levelname)s:%(message)s', level=logging.DEBUG)
    except:
        print("Error: Could not create log file! Exiting...")
        exit(1)

def assert_no_pipe_exists():
    if os.path.exists(IN_PIPE):
        logging.critical("Cannot start: Pipe file \"" + IN_PIPE + "\" already exists!")
        exit(1)

def make_pipe():
    try:
        os.mkfifo(IN_PIPE)
    except:
        logging.critical("Cannot start: Creating pipe file \"" + IN_PIPE + "\" failed!")
        exit(1)
    logging.debug("Created pipe \"" + IN_PIPE)

def read_from_pipe():
    try:
        buffer = os.read(inpipe, 255)
    except OSError as err:
        if err.errno == errno.EAGAIN or err.errno == errno.EWOULDBLOCK:
            buffer = None
        else:
            raise
 
    if buffer is None or len(buffer) == 0:
        logging.debug("Inpipe not ready.")
    else:
        logging.debug("Got data from the pipe: " + buffer.decode())
    
 # Main #
with daemon.DaemonContext(pidfile=daemon.pidfile.TimeoutPIDLockFile("/var/run/bdaemon.pid"), umask=0o002):
    signal.signal(signal.SIGTERM, handler_sigterm)
    start_logging()
    assert_no_pipe_exists()
    make_pipe()
    inpipe = os.open(IN_PIPE, os.O_RDONLY | os.O_NONBLOCK)
    logging.info("Baby Daemon started up and ready!")

    while True:
        time.sleep(5)
        read_from_pipe()

Apart from one more import, errno, we have three important changes here. First, the cleanup has been extended, there is a new function, called “read_from_pipe()” and then main has been modified as well. We’ll take a look at the latter first.

There’s a ton of examples on named pipes on the net, but they usually use just one program that forks off a child process and then communicates over the pipe with that. That’s pretty simple to do and works nicely by just copying and pasting the example code in a file. But adapting it for our little daemon does not work: The daemon just seems to “hang” after first trying to read something from the pipe. What’s happening there?

By default, reads from the pipe are in blocking mode, which means that on the attempt to read, the system just waits for data if there is none! The solution is to use non-blocking mode, which however means to use the raw os.open function (that supports flags to be passed to the operating system) instead of the nice Python open function with its convenient file object.

So what does the line starting with “inpipe” do? It calls the function os.open and tells it to open IN_PIPE where we defined the location of our pipe. Then it gives the flags, so that the operating system knows how to open the file, in this case in read-only and in non-blocking mode. We need to open it read-only, because the daemon should be at the receiving side of the pipe. And, yes, we want non-blocking, so that the program continues on if there is no data in the pipe without waiting for it all the time!

What might look a little strange to you, is the | character between the two flags. Especially since on the terminal it’s known as the pipe character and we’re talking about pipes here, right? In this case it’s something completely unrelated however. That symbol just happens to be Python’s choice for representing the bit-wise OR operator. Let’s leave it at that (I’ll explain a bit more of it in a future “Python pieces” section, but this article will be long enough without it).

However that’s still not all that the line we’re just discussing does. The os.open() function returns a file descriptor that we’re then assigning to the inpipe variable to keep it around.

What’s left is a new infinite loop that calls read_from_pipe() every 5 seconds.

Speaking of that function, let’s take a closer look at what it does. It tries to use the os.read function to read up to 255 bytes from the pipe into the variable named buffer. We’re doing so in a try/except block, because the read is somewhat likely to fail (e.g. if the pipe is empty). When there’s an exception, the code checks for the exact error that happened and if it’s EAGAIN or EWOULDBLOCK, we deliberately empty the buffer. If some other error occurred, it’s something that we didn’t expect, so let’s better take the straight way out by raising the exception again and crashing the program.

On FreeBSD the error numbers are defined in /usr/include/errno.h. If you take a look at it, you see that EAGAIN and EWOULDBLOCK are the same thing, so checking for one of them would be enough. But it makes sense to know that on some systems these are separate errors and that it’s good practice to check for both.

If the buffer either has the None value or has a length of 0, we assume that the read failed. Otherwise we put the data into the log. To make it readable we have to use decode, because we will be receiving encoded data.

All that’s left is the cleanup function. I’ve added another try/except block that simply tries to close the pipe file before trying to delete it. This is example code, so to make things not even more complex, I just silently ignore if the attempt fails.

Control script

Ok, great! That was quite a bit of things to cover, but now we have a daemon that creates a pipe and tries to read data from it. There’s just one problem: How can we test it? By creating another, separate program, that puts data in the pipe of course! For that let’s create another file with the name bdaemonctl.py:

#!/usr/local/bin/python3.6

 # Imports #
import os, time

 # Globals #
OUT_PIPE = '/var/run/bd_in.pipe'

 # Main #
try:
    outpipe = os.open(OUT_PIPE, os.O_WRONLY)
except:
    raise

for i in range(0, 21):
    print(i)
    try:
        os.write(outpipe, bytes(str(i).encode('utf-8')))
    except BrokenPipeError:
        print("Pipe has disappeared, exiting!")
        os.close(outpipe)
        exit(1)
    time.sleep(3)
os.close(outpipe)

Fortunately this one is fairly simple. We do our imports and define a variable for the pipe. We could skip the latter, because we’re using it on only one occasion but in general it’s a good idea to keep it as it is. Why? Because hiding things deep in the code may not be such a smart move. Defining things like this at the top of the file increases the maintainability of your code a lot. And since we want to send data this time, of course we name our variable OUT_PIPE appropriately.

In the main section we just try to open the pipe file and crash if that doesn’t work. It’s pretty obvious that such a case (e.g. the pipe is not there because the daemon is not running) should be better handled. But I wanted to keep things simple here because it’s just an example after all.

Then we have a loop that counts from 0 to 20, outputs the current number to stdout and tries to also send the data down the pipe. If that works, the program waits three seconds and then continues the loop.

To be able to write to the pipe we need a byte stream but we only have numbers. We first convert them to a string and use a proper encoding (utf8) and then convert them to bytes that can be sent over the pipe.

When the loop is over, we close the pipe file properly because we as the sender are done with it. I added a little bit of code to handle the case when the daemon exits while the control script runs and still tries to send data over the pipe. This results in a “broken pipe” error. If that happens, we just print an error message, close the file (to not leak the file descriptor) and exit with an error code of 1.

So for today we’re done! We can now send data from a control program to the daemon and thus have achieved uni-directional communication between two processes.

What’s next?

I’ll take a break from these programming-related posts and write about something else next.

However I plan to continue with a 4th part later which will cover argument parsing. With that we could e.g. modify our control program to send arbitrary data to the daemon from the command line – which would of course be much more useful than the simple test case that we have right now.

Writing a daemon using FreeBSD and Python pt.2

The previous part of this series left off with a running “baby daemon” example. It covered Python fundamentals, signal handling, logging as well as an init script to start the daemon.

Daemonization with Python

The outcome of part 1 was a program that needed external help actually to be daemonized. I used FreeBSD’s handy daemon(8) utility to put the program into the background, to handle the pidfile, etc. Now we’re making one step forward and try to achieve the same thing using just Python.

To do that, we need a module that is not part of Python’s standard library. So you might need to first install the package py36-daemon if you don’t already have it on your system. Here’s a small piece of code for you – but don’t get fooled by the line count, there’s actually a lot of things going on there (and of concepts to grasp):

#!/usr/local/bin/python3.6
 
 # Imports #
import daemon, daemon.pidfile
import logging
import signal
import time

 # Fuctions #
def handler_sigterm(signum, frame):
    logging.debug("Exiting on SIGTERM")
    exit(0)

def main_program():
    signal.signal(signal.SIGTERM, handler_sigterm)
    try:
        logging.basicConfig(filename='/var/log/bdaemon.log', format='%(levelname)s:%(message)s', level=logging.DEBUG)
    except:
        print("Error: Could not create log file! Exiting...")
        exit(1)
 
    logging.info("Started!")
    while True:
        time.sleep(1)

 # Main #
with daemon.DaemonContext(pidfile=daemon.pidfile.TimeoutPIDLockFile("/var/run/bdaemon.pid"), umask=0o002):
    main_program()

I dropped some ballast from the previous version; e.g. overriding SIGINT was a nice thing to try out once, but it’s not useful as we move on. Also that countdown is gone. Now the daemon continues running until it’s signaled to terminate (thanks to what is called an “infinite loop”).

We have two new imports here that we need for the daemonization. As you can see, it is possible to import multiple modules in one line. For readability reasons I wouldn’t recommend it in general. I only do it when I import multiple modules that kind of belong together anyway. However in the coming examples I might just put everything together to save some lines.

The first more interesting thing with this version is that the main program was moved to a function called “main_program”. We could have done that before if we really wanted to, but I did it now so the code doesn’t take attention away from the primary beast of this example. Take a look at the line that starts with the with keyword. Now that’s a mouthful, isn’t it? Let’s break this one up into a couple of pieces so that it’s easier to chew, shall we?

The value for umask is looking a bit strange. It contains an “o” among the numbers, so it has to be a string, doesn’t it? But why is it written without quotes then? Well, it is a number. Python uses the “0o” prefix to denote octal (the base-8 numbering system) numbers and 0x would mean hexadecimal (base-16) ones.

Remember that we talked about try/except before (for the logging)? You can expand on that. A try block can not only have except blocks, it can also have a finally block. Statements in such a block are meant to be executed no matter the outcome of the try block. The classical example is that when you open a file, you definitely want to close it again (everything else is a total mess and would make your program an exceptionally bad one).

Closing it when you are done is simple. But what if an exception is raised? Then the code path that properly closes the file might never be reached! You could close the file in every thinkable scenario – but that would be both tedious and error-prone. For that reasons there’s another way to handle those cases: Close the file in the finally block and you can be sure that it will be closed regardless of what happens in the try or in any except block.

Ok, but what does this have to do with our little daemon? Actually a lot. That case of try/finally has been so common that Python provides a shortcut with so-called context managers. They are objects that manage a resource for you like this: You request it, it is valid only inside one block (the with one!) and when the block ends, the context manager takes care of properly cleaning up for you without having you add any extra code (or even without you knowing, if you just copy/paste code from the net without reading explanations like this).

So the with statement in our code above lets Python handle the daemonization process while the main_program function is running. When it ends on the signal, Python cleans up everything and the process terminates – which is great for us. Accept that for now and live with the fact that you might not know just how it does that. We’ll come back to things like that.

Updated init script

Ok, the one thing left to do here is making the required changes to the init script. We are no longer using the daemon(8) utility, so we need to adjust it. Here it is the new one:

#!/bin/sh

. /etc/rc.subr

name=bdaemon
rcvar=bdaemon_enable

command="/root/bdaemon.py"
command_interpreter=/usr/local/bin/python3.6
pidfile="/var/run/${name}.pid"

load_rc_config $name
run_rc_command "$1"

Not too much changed here, but let’s still go over what has. The command definition is pretty obvious: The program can now daemonize itself, so we call it directly. It doesn’t take any arguments, which means we can drop command_args.

However we need to add command_interpreter instead (one important thing that I had overlooked first), because the program will look like this in the process list:

/usr/local/bin/python3.6 /root/bdaemon.py

Without defining the interpreter, the init system would not recognize this process as being the correct one. Then we also need to point it to the to the pidfile, because in theory there could be multiple processes that match otherwise.

And that’s it! Now we have a daemon process running on FreeBSD, written in pure Python.

Python pieces

This next part is a completely optional excursion for people who are pretty new to programming. We’ll take a step back and discuss concepts like functions and arguments, modules, as well as namespaces. This should help you better understand what’s happening here, if you like to know more. Feel free to save some time and skip the excursion if you are familiar with those things.

Functions and arguments

As you’ve seen, functions are defined in Python by using the def keyword, the function name and – at the very least – an empty pair of parentheses. Inside the parentheses you could put one or more arguments if needed:

def greet(name):
    print("Hi, " + name + "!")

greet("Alice")
greet("Bob")

Here we’re passing a string to the function that it uses to greet that person. We can add a second argument like this:

def greet(name, phrase):
    print("Hi, " + name + "! " + phrase)

greet("Alice", "Great to see you again!")
greet("Bob", "How are you doing?")

The arguments used here are called positional arguments, because it’s decided by their position what goes where. Invert them when calling the function and the output will obviously be garbage as the strings are assigned to the wrong function variable. However it’s also possible to refer to the variable by name, so that the order does no longer matter:

def greet(name, phrase):
    print("Hi, " + name + "! " + phrase)

greet(phrase="Great to see you again!", name="Alice")
greet("Bob", "How are you doing?")

This is what is used to assign the values for the daemon context. Technically it’s possible to mix the ways of calling (as done here), but that’s a bit ugly.

We’re not using it, yet, but it’s good to know that it exists: There are also default values. Those mean that you can leave out some arguments when calling a function – if you are ok with the default value.

def greet(name, phrase = "Pleased to meet you."):
    print("Hi, " + name + "! " + phrase)

greet(phrase="Great to see you again!", name="Alice")
greet("Bob", "How are you doing?")
greet("Carol")

And then there’s something known as function overloading. We’re not going into the details here, but you might want to know that you can have multiple functions with the same name but a different number of arguments (so that it’s still possible to precisely identify which one needs to be called)!

Modules

When reading about Python it usually won’t take too long before you come across the word module. But what’s a module? Luckily that’s rather easy to explain: It’s a file with the .py extension and with Python code in it. So if you’ve been following this daemon tutorial, you’ve been creating Python modules all the way!

Usually modules are what you might want to refer as to libraries in other languages. You can import them and they provide you with additional functions. You can either use modules that come with Python by default (that collection of modules is known as the standard library, so don’t get confused by the terminology there), additional third-party modules (there are probably millions) or modules that you wrote yourself.

It’s fairly easy to do the latter. Let’s pick up the previous example and put the following into a file called “greeter.py”:

forgot_name = "Sorry, what was your name again?"

def greet(name, phrase = "Pleased to meet you."):
    print("Hi, " + name + "! " + phrase)

Now you can do this in another Python program:

import greeter

greeter.greet("Carol")
print(greeter.forgot_name)

This shows that after importing we can use the “greet()” function in this program, even though it’s defined elsewhere. We can also access variables used in the imported module (greeter.forgot_name in this case).

Namespaces

Ever wondered what that dot means (when it’s not used in a filename)? You can think of it as a hierarchical separator. The standard Python functions (e.g. print) are available in the global namespace and can thus be used directly. Others are in a different namespace and to use them, it’s necessary to refer to that namespace as well as the function name so that Python understand what you want and finds the function. One example that we’ve used is time.sleep().

Where does this additional namespace come from? Well, remember that we did import time at the top of the program? That created the “time” namespace (and made the functions from the time module available there).

There’s another way of importing; we could import either everything (using an asterisk (*) character, but that’s considered poor coding) or just specific functions from one module into the global namespace:

from time import sleep
sleep(2)
exit(0)

This code will work because the “from MODULE import FUNCTION” statement in this example imported the sleep function so that it becomes available in the global namespace.

So why do we go through all the hassle to have multiple namespaces in the first place? Can’t we just put everything in the global one? Sure, we could – and for more simple programs that’s in fact an option. But consider the following case: Python provides the open keyword. It’s used to open a file and get a nice object back that makes accessing or manipulating data really easy. But then there’s also os.open, which is not as friendly, but let’s you use more advanced things since it uses the raw operating system functionality. See the problem?

If you import the functions from os into the global namespace, you have a name clash in the case of open. This is not an error, mind you. You can actually do that, but you should know what happens. The function imported later will override the one that went by that name previously, effectively making the original one inaccessible. This is called “shadowing” of the original function.

To avoid problems like this it’s often better to have your own separate namespace where you can be sure that no clashes happen.

What’s next?

In the next part we’ll take a look at implementing IPC (inter-process communication) using named pipes (a.k.a “fifos”).

Writing a daemon using FreeBSD and Python pt.1

Being a sysadmin by profession, I don’t code. At least not often enough or with as high quality output that programmers would accept to call coding. I do write and maintain shell scripts. I also write new formulas for configuration management with SaltStack.

The latter is Python-based and after hearing mostly good things about that language, I’ve been trying to do some simple things with it for a while now. And guess what: It’s just so much more convenient compared to using shell code! I’ll definitely keep doing some simple tasks in Python, just to get some experience with it.

Not too long I thought about a little project that I’d try to do and decided to go with Python again. Thinking about what the program should do, I figured that a daemon would make a nice fit for it. But how do you write a daemon? Fortunately it’s especially easy on FreeBSD. So let’s go!

Python

The first thing that I did, was to create a new file called bdaemon.py (for “baby daemon”) and use chmod to make it executable. And here’s what I put into it as a first test:

#!/usr/local/bin/python3.6

 # Imports #
import time

 # Globals #
TTL_SECONDS = 30
TTL_CHECK_INTERVAL = 5

 # Fuctions #

 # Main #
print("Started!")
for i in range(1, TTL_SECONDS + 1):
    time.sleep(1)
    if i % TTL_CHECK_INTERVAL == 0:
        print("Running for " + str(i) + " seconds...")
print("TTL reached, terminating!")
exit(0)

This very simple program has the shebang line that points the operating system to the right interpreter. Then I import Python’s time module which gives me access to a lot of time-related functions. Next I define two global variables that control how long the program runs and in which interval it will give output.

The main part of the program first outputs a starting message on the terminal. It then enters a for loop, that counts from 1 to 30. In Python you do this by providing a list of values after the in keyword. Counting to 5 could have been written as for i in [1, 2, 3, 4, 5]: for example.

With range we can have Python create a list of sequential numeric values on the fly – and since it’s much less to type (and allows for dynamic list creation by setting the final number via a variable), I chose to go with that. Oh, BTW: In Python the last value of those ranges is exclusive, not inclusive. This means that range(1, 5) leads to [1, 2, 3, 4] – if you want the 5 included in the list, you have to use range(1, 6)! That’s why I add 1 to the TTL_SECONDS variable.

I use time.sleep to create a delay in the loop block. Then I do a check if the remainder of the division of the current running time by the defined check interval is zero (% is the modulus operator which gives that remainder value of the division). If it is, the program creates more output.

Mind the indentation: In Python it is used to create code blocks. The for statement is not indented, but it ends with a colon. That means that it’s starting a code block. Everything up to (but not including) the second to last print statement is indented by four spaces and thus part of the code block. Said print statement is indented two levels (8 spaces) – that’s because it’s another block of its own started by the if statement before it. We could create a third, forth and so on level deep indentation if we required other blocks beneath the if block.

Eventually the program will print that the TTL has been reached and exit the program with an error code of 0 (which means that there was no error).

Have you noticed the str(i) part in one of the print statements? That is required because the counter variable “i” holds numeric values and we’re printing data of a different type. So to be able to concatenate (that’s what the plus sign is doing in this case!) the variable’s contents to the rest of the data, it needs to match its type. We’re achieving this by doing a conversion to a string (think converting the number 5 to the literal “5” that can be part of a line of text where it looks similar but is actually a different thing).

Oh, and the pound signs are used to start comments that are ignored by Python. And that’s already it for some fundamental Python basics. Hopefully enough to understand this little example code (if not, tell me!).

Signals

The next thing to explore is signal handling. Since a daemon is essentially a program running in the background, we need a way to tell it to quit for example. This is usually done by using signals. You can send some of them to normal programs running in the terminal by hitting key combinations, while all of them can be sent by the kill command.

If you press CTRL-C for example, you’re sending SIGINT to the currently running application, telling it “abort operation”. A somewhat similar one is SIGTERM, which kind of means “hey, please quit”. It’s a graceful shutdown signal, allowing the program to e.g. do some cleanup and then shut down properly.

If you use kill -9, however, you’re sending SIGKILL, the ungraceful shutdown signal, that effectively means “die!” for the process targeted (if you’ve ever done that to a live database or another touchy application, you know that you really have to think before using it – or you might be in for all kinds of pain for the next few hours).

#!/usr/local/bin/python3.6

 # Imports #
import signal
import time

 # Globals #
TTL_SECONDS = 30
TTL_CHECK_INTERVAL = 5

 # Fuctions #
def signal_handler(signum, frame):
    print("Received signal" + str(signum) + "!")
    if signum == 2:
        exit(0)

 # Main #
signal.signal(signal.SIGHUP, signal_handler)
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGQUIT, signal_handler)
signal.signal(signal.SIGILL, signal_handler)
signal.signal(signal.SIGTRAP, signal_handler)
signal.signal(signal.SIGABRT, signal_handler)
signal.signal(signal.SIGEMT, signal_handler)
#signal.signal(signal.SIGKILL, signal_handler)
signal.signal(signal.SIGSEGV, signal_handler)
signal.signal(signal.SIGSYS, signal_handler)
signal.signal(signal.SIGPIPE, signal_handler)
signal.signal(signal.SIGALRM, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)
#signal.signal(signal.SIGSTOP, signal_handler)
signal.signal(signal.SIGTSTP, signal_handler)
signal.signal(signal.SIGCONT, signal_handler)
signal.signal(signal.SIGCHLD, signal_handler)
signal.signal(signal.SIGTTIN, signal_handler)
signal.signal(signal.SIGTTOU, signal_handler)
signal.signal(signal.SIGIO, signal_handler)
signal.signal(signal.SIGXCPU, signal_handler)
signal.signal(signal.SIGXFSZ, signal_handler)
signal.signal(signal.SIGVTALRM, signal_handler)
signal.signal(signal.SIGPROF, signal_handler)
signal.signal(signal.SIGWINCH, signal_handler)
signal.signal(signal.SIGINFO, signal_handler)
signal.signal(signal.SIGUSR1, signal_handler)
signal.signal(signal.SIGUSR2, signal_handler)
#signal.signal(signal.SIGTHR, signal_handler)

print("Started!")
for i in range(1, TTL_SECONDS + 1):
    time.sleep(1)
    if i % TTL_CHECK_INTERVAL == 0:
        print("Running for " + str(i) + " seconds...")
print("TTL reached, terminating!")
exit(0)

For this little example code I’ve added a function called “signal_handler” – because that’s what it is for. And in the main program I installed that signal handler for quite a lot of signals. To be able to do that, I needed to import the signal module, of course.

If this program is run, it will handle every signal you can send on a FreeBSD system (run kill -l to list all available signals on a Unix-like operating system). Why are some of those commented out? Well, try commenting those lines in! Python will complain and stop your program. This is because not all signals are allowed to be handled.

SIGKILL for example by its nature is something that you don’t want to allow to be overridden with custom behavior after all! While your program can choose to handle e.g. SIGINT and choose to ignore it, SIGKILL means that the process totally needs to be shutdown immediately.

Try running the program and send some signals while it’s running. On BSD systems you can e.g. send CTRL-T for SIGINFO. The operating system prints some information about the current load. And then the program has the chance to output some additional information (some may tell you what file they currently process, how much percent they have finished copying, etc.). If you send SIGINT, this program terminates as it should.

Logging

There’s another thing that we have to consider when dealing with processes running in the background: A daemon detaches from the TTY. That means it can no longer receive input the usual way from STDIN. But we investigated signals so that’s fine. However it also means a daemon cannot use STDOUT or STDERR to print anything to the terminal.

Where does the data go that a daemon writes to e.g. STDOUT? It goes to the system log. If no special configuration for it exists, you will find it in /var/log/messages. Since we expect quite a bit of debug output during the development phase, we don’t really want to clutter /var/log/messages with all of that. So to write a well-behaving little daemon, there’s one more topic that we have to look into: Logging.

#!/usr/local/bin/python3.6

 # Imports #
import logging
import signal
import time

 # Globals #
TTL_SECONDS = 30
TTL_CHECK_INTERVAL = 5

 # Fuctions #
def handler_sigterm(signum, frame):
    logging.debug("Exiting on SIGTERM")
    exit(0)

def handler_sigint(signum, frame):
    logging.debug("Not going to quit, there you have it!")

 # Main #
signal.signal(signal.SIGINT, handler_sigint)
signal.signal(signal.SIGTERM, handler_sigterm)
try:
    logging.basicConfig(filename='bdaemon.log', format='%(levelname)s:%(message)s', level=logging.DEBUG)
except:
    print("Error: Could not create log file! Exiting...")
    exit(1)

logging.info("Started!")
for i in range(1, TTL_SECONDS + 1):
    time.sleep(1)
    if i % TTL_CHECK_INTERVAL == 0:
        logging.info("Running for " + str(i) + " seconds...")
logging.info("TTL reached, terminating!")
exit(0)

The code has been simplified a bit: Now it installs only handlers for two signals – and we’re using two different handler functions. One overrides the default behavior of SIGINT with a dummy function, effectively refusing the expected behavior for testing purposes. The other one handles SIGTERM in the way it should. If you are fast enough on another terminal window, you can figure out the PID of the running program and then kill -15 it.

Logging with Python is extremely simple: You import the module for it, call a function like logging.basicConfig – and start logging. This line sets the filename of the log to “bdaemon.log” (for “baby daemon”) in the current directory. It changes the default format to displaying just the log level and the actual message. And then it defines the lowest level that should be logged.

There are various pre-defined levels like debug, info, warning, critical, etc. But what’s that try and except thing? Well, the logging module will attempt to create a logfile (or append to it, if it already exists). This is an operation that could fail. Perhaps we’re running the program in a directory where we don’t have the permission to create the log file? Or maybe for whatever reason a directory of that name exists? In both cases Python cannot create the file an an error occurs.

If such a thing happens, Python doesn’t know what to do. It knows what the programmer wanted to do, but has no clue on what to do if things fail. Does it make sense to keep the program running if something unexpected happened? Probably not. So it throws an exception. If an unhandled exception occurs, the program aborts. But we can catch the exception.

By putting the function that opens the file in a try block, we’re telling Python that we’re expecting it could fail. And with except we can catch an exception and handle expected problems. There are a lot of exception types; by not specifying any, we’re catching all of them. That might not be the best idea, because maybe something else happened and we’re just expecting that the logfile could not be created. But let’s keep it simple for now.

The one remaining thing to do is to change any print statements so that we’re using the logging instead. Depending on how important the log entry is, we can also use different levels from least important (DEBUG) to most important (CRITICAL).

You can either wait for the program to finish and then take a look at the log, or you open a second terminal and tail -f bdaemon.log there to watch output as the program is running.

Alright! With this we have everything required to daemonize the program next. Let’s write a little init script for it, shall we?

Init

Init scripts are used to control daemons (start and stop them, telling them to reload the configuration, etc.). There are various different init systems in use across the Unix-like operating system. FreeBSD uses the standard BSD init system called rc.d. It works with little (or not so little if you need to manage very complex daemons) shell scripts.

Since a lot of the functionality of the init system is all the same across most of these scripts, rc.d handles all the common cases in shell files of it’s own that are then used in each of the scripts. In Python this would be done by importing a module; the term with shell scripting is to source another shell script (or fragment).

Create the file /usr/local/etc/rc.d/bdaemon with the following contents:

#!/bin/sh

. /etc/rc.subr

name=bdaemon
rcvar=bdaemon_enable

command="/usr/sbin/daemon"
command_args="-p /var/run/${name}.pid /path/to/script/bdaemon.py"

load_rc_config $name
run_rc_command "$1"

Yes, you need root privileges to do that. Daemons are system services and so we’re messing with the system now (totally at beginner level, though). Save the file and you should be able to start the program as a daemon e.g. by running service bdaemon onestart!

How’s that? What does that all mean and where does the daemonization happen? Well, the first line after the shebang sources the main rc fragment with all the required functions (read the dot as “source”). Then it defines a name for the daemon and an rcvar.

What is an rcvar? Well, by putting “bdaemon_enable=YES” into your /etc/rc.conf you could enable this daemon for automatic startup when the system is coming up. If that line is not present there, the daemon will not start. That’s why we need to use “onestart” to start it anyway (try it without the “one” if you’ve never done that and see what happens!).

Then the command to run as well as the arguments for that command are defined. And eventually two helper functions from rc.subr are called which do all the actual complex magic that they thankfully hide from us!

Ok, but what is /usr/sbin/daemon? Well, FreeBSD comes with an extremely useful little utility that handles the daemonization process for others! This means it can help you if you want to use something as a background service but you don’t want to handle the actual daemonization yourself. Which is perfect in our case! With it you could even write a daemon in shell script for example.

The “-p” argument tells the daemon utility to handle the PID file for the process as well. This is required for the init system to control the daemon. While our little example program is short-lived, we can still do something while it runs. Try out service onestatus and service onestop on it for example. If there was no PID file present, the init system would claim that the daemon is not running, even if it is! And it would not be able to shut it down.

There we go. Our first FreeBSD daemon process written in Python! One last thing that you should do is change the filename for the logfile to use an absolute path like /var/log/bdaemon.log. If you want to read more about the daemon utility, read it’s manpage, daemon(8). And should you be curious about what the init system can do, have a look here.

What’s next?

While using /usr/sbin/daemon is perfectly fine, you might feel that we kind of cheated. So next time we’ll take a brief look at daemonizing with Python directly.

I also want to explore IPC (“inter-process communication) with named pipes. This will allow for a little bit more advanced daemon that can be interacted with using a separate program.