Multi-OS PXE-booting from FreeBSD 12: PXE menu and *BSD (pt. 3)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://

Post 1 of this mini series is about what lead me to do this in the first place, features a little excursion for people new to PXE and details the setup of a FreeBSD router.
Post 2 discusses setting up the required daemons for DHCP, TFTP and HTTP / FTP. Each component that is not in FreeBSD’s base system is briefly discussed and two options to pick from are covered.

At the end of part 2, the situation is as follows: A client in the subnet attempting to PXE boot will get an IP address via DHCP and be told where to get the NPB (Network Bootstrap Program). It will then attempt to fetch it via TFTP. There’s just one problem: That is not there, yet! We’ll fix that in a minute. When we’re done here, you’ll have a fully working PXE server that offers multiple BSD operating systems to boot into (Linux and more are covered in part 4). This article focuses on BIOS (“legacy”) booting; if you want to boot EFI-only machines you’ll have to adapt the configuration examples given here to that. I also assume using HTTP here – if you opted for FTP you will have to adapt the commands used in the examples.

Network Bootstrap Program

There are a lot of NBPs available. Usually each operating system has its own which is tuned towards its specific duty: FreeBSD has one, OpenBSD has another and Linux has several. These are not ordinary programs; they need to cope with a very resource-constrained environment and cannot depend on any external libraries. While writing boot code is challenging enough, adding network booting capabilities doesn’t make things any easier. This is why most NBPs are as simple as possible.

As a result of that, the NBPs usually know how to boot exactly one operating system. Since we want to set up a multi-OS PXE server this is quite unfortunate for our use case. There are two ways to work around this problem:

  1. Provide various NBPs and use DHCP to distinguish between clients
  2. Use an NBP that supports a menu to select which one to boot next

As usual there’s pros and cons to both. Letting DHCP do the magic requires a much more complex DHCP configuration. It’s also much less flexible. The boot menu approach is simple and flexible, but more complicated if you are also interested in automation. I do like automation, but I decided in favor of using a boot menu for this article because it’s easier to follow. It is also a good achievement to build upon once you’re comfortable with DHCP and feel ready for advanced settings.

It is possible to use one NBP to fetch and execute another one. This process is known as chainloading. For some operating systems that is the best choice to add them to a menu system.

There’s three popular options that we have for an NBP which fits our needs:

2. PXELINUX (from Syslinux) and
3. iPXE

GRUB and I never made friends. I used it for a while after switching from LILO only to ditch it for Syslinux when I learned of that. I have to use it on many systems, but when I have a choice, I choose something else. The iPXE project is very interesting. It’s the most advanced (but also most involved) of the three options. If you’re curious about just how far you can take PXE booting, at least have a look at it. For this article, we’ll go with PXELINUX.


Pxelinux is available via packages on FreeBSD. It does pull in some dependencies that I don’t want on my system however. For that reason we’re going to fetch the package instead of installing it. Then we extract it manually:

# pkg fetch -y syslinux
# mkdir /tmp/syslinux
# tar -C /tmp/syslinux -xvf /var/cache/pkg/syslinux-6.03.txz

Now we can cherry-pick the required files:

# cp /tmp/syslinux/usr/local/share/syslinux/bios/core/lpxelinux.0 /usr/local/tftpboot/pxelinux.0
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/elflink/ldlinux/ldlinux.c32  /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/menu/vesamenu.c32 /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/lib/libcom32.c32 /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/libutil/libutil.c32 /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/modules/pxechn.c32 /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/memdisk/memdisk /usr/local/tftpboot/
# rm -r /tmp/syslinux

The first one is the modular NBP itself. It requires some modules – the c32 files. The pxechn and memdisk modules are optional – they are required for some of the operating systems examples here but not all. You can leave them out if you don’t need them. Restart the inetd service now and you will be able to PXE boot to the menu:

# service inetd restart

Keep in mind: Restart the inetd service whenever you added a file to or edited any in the tftpboot directory!

Tip 1: You can use make use of gzipped files as Pxelinux supports that. This way you can decrease loading times by serving smaller images over the net.

Tip 2: I’m using gzip in my examples but if you really want to fight for the last byte, use zopfli instead. It’s a compression program that produces gzip-compatible output but takes much more processor time to create optimized archives. As decompression time is unaffected it’s a price you have to pay only once. Consider using it if you want the best results.


Pxelinux is hard-coded to load pxelinux.cfg/default via TFTP and use that as its configuration. If you plan to only use few of the OS examples shown here, that config is sufficient as you can put everything into there. Once you feel that your menu is becoming overly crowded, you can turn it into a main menu and make use of submenus to group things as I do it here. This works by putting the various OS entries in different config files.

If you don’t want submenus, skip the next step and put all the menu entries that go into something other than pxelinux.cfg/default in my examples right into that file instead – and leave out the reference back to the main menu since they don’t make any sense if you’re using a flat menu anyway.

In the previous post we already created the file /usr/local/tftpboot/pxelinux.cfg/default. Append the following to it to create a submenu for the BSDs we’re covering today:

MENU TITLE PXE Boot Menu (Main)

LABEL bsd-oses
        MENU LABEL BSD Operating Systems
        KERNEL vesamenu.c32
        APPEND pxelinux.cfg/bsd

Now create the file referenced there:

# vi /usr/local/tftpboot/pxelinux.cfg/bsd

and put the following text in there:


LABEL main-menu
        MENU LABEL Main Menu
        KERNEL vesamenu.c32
        APPEND pxelinux.cfg/default

Alright, preparation work is done, let’s finally add some operating system data!

FreeBSD 12.2

PXE booting FreeBSD is actually not such an easy thing to do if you want to avoid using NFS shares. Fortunately there is mfsBSD, a project that provides tools as well as releases of special FreeBSD versions that can be booted over the net easily. We’re going to use that.

There are multiple variants for download: The standard one, the “special edition” and a “mini edition”. The special edition comes with the distribution tarballs on the ISO – you may want to use that one for installation purposes. If you just want a FreeBSD live system (e.g. for maintenance and repairs) or use you owr mirror (see below), the standard edition is for you since it is much smaller and thus boots way faster.

Let’s make the image available via HTTP:

# mkdir -p /usr/local/www/pxe/bsd/fbsd
# fetch -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso
# gzip -9 /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso

Now edit the pxelinux.cfg/bsd file and append:

LABEL fbsd-pxe-install
        MENU LABEL Install FreeBSD 12.2 (PXE)
        KERNEL memdisk
        APPEND iso raw

That’s it. You can now PXE-boot into FreeBSD.

Installation using mfsBSD

Login with user root and password mfsroot. It includes a “zfsinstall” script that you may want to take a look at. There’s a lot more to mfsBSD, though. Its tools allow you to easily roll your own customized images. If a way to include packages or files, use a custom-built kernel and things like that sounds like something that would be useful for you, take a closer look. I cannot go into more detail here – it’s a topic of its own and would deserve an entire article dedicated to it. In case you just want to use the familiar FreeBSD installer bsdinstall, read on.

Mirroring distfiles and fixing bsdinstall

If you want to install FreeBSD over PXE more than once, it makes sense to provide a local distfile mirror. Since we have a fileserver running anyway, there’s really nothing more to it than getting the distfiles and putting them into the right place. At the very minimum get the following three files:

# fetch -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/MANIFEST
# fetch -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/base.txz
# fetch -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/kernel.txz

Depending on which distfiles you usually install, also get any or all of the following files:

# fetch -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/base-dbg.txz
# fetch -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/kernel-dbg.txz
# fetch -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/lib32-dbg.txz
# fetch -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/ports.txz
# fetch -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/src.txz
# fetch -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/tests.txz

At this point in time, bsdinstall is broken on mfsBSD. The reason is that the distfile manifest is missing. I think about getting this fixed upstream, so in the future try and see if the following part is obsolete before using it. But for now, let’s create a simple shell script in the webroot directory for convenience:

# vi /usr/local/www/pxe/

Put the following into the script:

ARCH=`uname -m`
RELEASE=`uname -r | cut -d"-" -f1`
mkdir -p /usr/freebsd-dist
fetch${ARCH}/${RELEASE}-RELEASE/MANIFEST -o /usr/freebsd-dist/MANIFEST

Now if you PXE-booted mfsBSD and logged in as root, you just need to execute the following command line and will then be able to use the installer as you are used to it:

# fetch && sh

When you are to select the installation source, there is an “Other” button at the bottom of the window. Choose that and point to your distfile mirror – in my example Happy installing!

One more hint: You may want to look into the environment variables that bsdinstall(8) provides. I briefly attempted to automatically set the URL to the distfile mirror but couldn’t get it working. As I was already running out of time with this article I haven’t looked deeper into it. If anybody figures it out I’d appreciate sharing your solution here.

OpenBSD 6.8

Adding OpenBSD as an option is trivial. The project provides a ramdisk kernel used for installing the system and a NBP capable of loading it. Let’s get those two files in place – and while the ramdisk kernel is fairly small already, we can chop a couple of bytes off by compressing it:

# mkdir -p /usr/local/tftpboot/bsd/obsd
# fetch -o /usr/local/tftpboot/bsd/obsd/pxeboot
# fetch -o /usr/local/tftpboot/bsd/obsd/6.8-amd64.rd
# gzip -9 /usr/local/tftpboot/bsd/obsd/6.8-amd64.rd

Now we only need to add the required lines to pxelinux.cfg/bsd:

LABEL obsd-pxe-install
        MENU LABEL Install OpenBSD 6.8 (PXE)
        KERNEL pxechn.c32
        APPEND bsd/obsd/pxeboot

That’s it, the OpenBSD loader can be booted! Since we don’t have the kernel in the assumed default location (“/bsd”) we’d need to tell the loader to “boot bsd/obsd/6.8-amd64.rd.gz”. The loader supports a configuration file, though. So for a little extra convenience we can make it pick up the kernel automatically like this:

# mkdir -p /usr/local/tftpboot/etc
# ln -s /usr/local/tftpboot/etc/boot.conf /usr/local/tftpboot/bsd/obsd/boot.conf
# echo "boot bsd/obsd/6.8-amd64.rd.gz" > /usr/local/tftpboot/bsd/obsd/boot.conf
# echo "# OpenBSD boot configuration" >> /usr/local/tftpboot/bsd/obsd/boot.conf

The pxeboot program comes with the configuraton file name of etc/boot.conf hard-coded. To keep things a little cleaner in the hierarchy that I use, I chose to set a symlink in the obsd directory for reference purposes. And that’s all.

NetBSD 9.1

Let’s add NetBSD! It’s somewhat similar to OpenBSD – but a bit more involved unfortunately. The reason is that the NBP by default does not support a configuration file. It has the ability to use one, but that needs to be activated first. Which is fair enough since it’s only a single command – on NetBSD that is! But let’s worry about this in a minute and first get the NBP as well as the install kernel:

# mkdir -p /usr/local/tftpboot/bsd/nbsd
# fetch -o /usr/local/tftpboot/bsd/nbsd/pxeboot_ia32.bin
# fetch -o /usr/local/tftpboot/bsd/nbsd/netbsd-INSTALL.gz

Now we need to add the boot menu entry by adding the following lines to pxelinux.cfg/bsd:

LABEL nbsd-pxe-install
        MENU LABEL Install NetBSD 9.1 (PXE)
        KERNEL pxechn.c32
        APPEND bsd/nbsd/pxeboot_ia32.bin

This is sufficient to load and execute the NetBSD loader. That will then complain that it cannot find the kernel and no hints about NFS were given. Now we have three options:

  1. Manually point the loader to the correct kernel each time
  2. Give the required hint via DHCP
  3. Try to enable the loader configuration

Typing in “tftp:bsd/nbsd/netbsd-INSTALL.gz” is probably fair enough if you are doing NetBSD installs very rarely but it gets old quickly. So let’s try out option two!

Modifying DHCP config for NetBSD

The DHCP server needs to be configured to pass a different Boot File name option when answering the NetBSD loader than otherwise. This is done by matching class information. This topic is beyond the scope of this article, so if you are interested, do some reading on your own. I won’t leave you hanging, though, if you just need to get things working.

Here’s what you have to add to the configuration if you’re using Kea – for example right before the “loggers” section:

    "client-classes": [
            "name": "NetBSDloader",
            "test": "option[vendor-class-identifier].text == 'NetBSD:i386:libsa'",
            "boot-file-name": "tftp:bsd/nbsd/netbsd-INSTALL.gz"

And here the same thing if you are using DHCPd:

if substring (option vendor-class-identifier, 0, 17) = "NetBSD:i386:libsa" {
    if filename = "netbsd" {
        filename "tftp:bsd/nbsd/netbsd-INSTALL.gz";

Restart your DHCP server and you should be good to go.

After accomplishing the easy way via DHCP, I also went down to explore the boot.cfg road but ultimately failed. I’m documenting it here anyway in case somebody wants to pick up where I decided to leave it be.

Enabling boot.cfg in the loader

To mitigate the risk of polluting my main system by doing something stupid I chose to do all of this using my unprivileged user. The first thing I did, was fetching and extracting the basic NetBSD 9.1 sources:

% mkdir -p netbsd-9.1 && cd netbsd-9.1
% fetch
% tar xvzf src.tgz

The sources for the installboot program we’re looking for are in usr/src/usr.sbin/installboot. I tried to get that thing to build by pointing the compiler at additional include directories and editing quite some header files, hoping to resolve the conflicts with FreeBSD’s system headers and problems like that. It can probably be done but that would take a C programmer – which I am not.

Fortunately NetBSD is the portability star among the BSDs and should be buildable on many other systems. I’ve never done this before but here was the chance. So I installed CVS and checked out the rest of the NetBSD src module:

% doas pkg install -y cvs
% cd netbsd-9.1/usr
% cvs -d checkout -r netbsd-9-1-RELEASE -P src

When exploring the source tree, I found a build script that obviously does all the magic required here. Be warned however that this builds a full cross-toolchain including the complete GCC compiler! Then it builds the “tools” subset of the NetBSD code (which includes the installboot that we’re looking for). On my old and slow Atom-based system this process took 6 hours:

% cd src
% ./ -U -m amd64 -T ~/nbsd tools
===> command:    ./ -U -m amd64 -T /home/kraileth/nbsd tools
===> started:    Thu Feb  3 00:13:41 CET 2021
===> NetBSD version:      9.1
===> MACHINE:             amd64
===> MACHINE_ARCH:        amd64
===> Build platform:      FreeBSD 12.2-RELEASE-p1 amd64
===> HOST_SH:             /bin/sh
===> No $TOOLDIR/bin/nbmake, needs building.
===> Bootstrapping nbmake
checking for sh... /bin/sh
checking for gcc... cc


install ===> config
#   install  /home/kraileth/nbsd/bin/nbconfig
mkdir -p /home/kraileth/nbsd/bin
/home/kraileth/nbsd/bin/amd64--netbsdelf-install -c  -r -m 555 config /home/kraileth/nbsd/bin/nbconfig
===> Tools built to /home/kraileth/nbsd
===> ended:      Thu Feb  3 6:26:25 CET 2021
===> Summary of results: command:    ./ -U -m amd64 -T /home/kraileth/nbsd tools started:    Thu Feb  4 07:13:41 CET 2021
         NetBSD version:      9.1
         MACHINE:             amd64
         MACHINE_ARCH:        amd64
         Build platform:      FreeBSD 12.2-RELEASE-p1 amd64
         HOST_SH:             /bin/sh
         No $TOOLDIR/bin/nbmake, needs building.
         Bootstrapping nbmake
         MAKECONF file:       /etc/mk.conf (File not found)
         TOOLDIR path:        /home/kraileth/nbsd
         DESTDIR path:        /usr/home/kraileth/netbsd-9.1/usr/src/obj/destdir.amd64
         RELEASEDIR path:     /usr/home/kraileth/netbsd-9.1/usr/src/obj/releasedir
         Created /home/kraileth/nbsd/bin/nbmake
         Updated makewrapper: /home/kraileth/nbsd/bin/nbmake-amd64
         Tools built to /home/kraileth/nbsd ended:      Thu Feb  4 12:26:25 CET 2021
===> .

The “-U” flag enables some trickery to build as an unprivileged user. With “-m” you specify the target architecture (I did use i386 but modified the above lines as that will be what most people will want to use instead!). Finally the “-T” switch allows to specify the installation target directory and the “tools” is the make target to use.

When it was done, I did the following (as root):

# cp /usr/local/tftpboot/bsd/nbsd/pxeboot_ia32.bin /usr/local/tftpboot/bsd/nbsd/pxeboot_ia32.bin.bak
# /usr/home/kraileth/netbsd-9.1/usr/src/tools/installboot/obj/installboot -eo bootconf /usr/local/tftpboot/bsd/nbsd/pxeboot_ia32.bin

This should enable the boot config file on the pxeboot loader. It does change the file and probably even makes the right change. I tried to enable module support via installboot, too and that obviously worked (the NFS module was loaded the next time I chainloaded the NetBSD loader). But for some reason I could not get boot.cfg to do what I wanted. Probably I don’t understand the file properly…

While it’s a bit disappointing to stop so close to the goal, messing with NetBSD so much already took much more time away from the other BSDs than I had imagined. And since I could at least offer a working way this was when I decided to move on.

DragonFly BSD

I attempted to get DragonFly BSD to work but failed. I briefly tried out a setup that includes NFS shares but it didn’t work completely either: Kernel booted but failed to execute /sbin/init for some reason or another. Also I don’t really want to cover NFS in this series – there’s enough material in here already. And without NFS… Well, DragonFly BSD has the same problem that FreeBSD has: It will boot the kernel but then be unable to mount the root filesystem.

While I guess that the mfsBSD approach could likely work for DF, too, this is something much more involved than reasonable for our topic here. I would really like to cover DragonFly here, too, but that’s simply a bit too much. If anybody knows how to get it working – please share your knowledge!

HardenedBSD 12-STABLE

HardenedBSD being a FreeBSD fork, inherited the same characteristics as vanilla FreeBSD. Which means that PXE booting the standard images is not an easy thing to do. HardenedBSD uses the same installer, bsdinstall however and for that reason it’s possible to install HardenedBSD by using mfsBSD as prepared above in the FreeBSD section. We only need to point the installer to a different distfile mirror. Let’s create that one now:

# mkdir -p /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE
# fetch -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/MANIFEST
# fetch -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/base.txz
# fetch -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/kernel.txz

As with FreeBSD, there are some optional distfiles you may or may not want to mirror, too. Provide what you need for your installations:

# fetch -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/base-dbg.txz
# fetch -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/kernel-dbg.txz
# fetch -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/src.txz
# fetch -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/tests.txz

Now we’re creating a convenience script for HardenedBSD:

# vi /usr/local/www/pxe/

Put the following into the script:

ARCH=`uname -m`
MAJOR=`uname -r | cut -d"." -f1`
mkdir -p /usr/freebsd-dist
fetch${ARCH}/${MAJOR}-STABLE/MANIFEST -o /usr/freebsd-dist/MANIFEST

Now fire up mfsBSD, login as root and simply run the following command line to start the installer:

# fetch && sh

Select the “Other” button when asked for the installation source. Choose that and point to your distfile mirror – in my example And that’s it.

MidnightBSD 2.0

Since MidnightBSD is a FreeBSD fork as well, it also suffers from the well-known problems related to PXE booting. Again mfsBSD comes to the rescue. Let’s create the distfile mirror first:

# mkdir -p /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE
# fetch -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/MANIFEST
# fetch -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/base.txz
# fetch -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/kernel.txz

Pick any or all of the remaining optional distfiles to be mirrored, too, if you need them:

# fetch -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/lib32.txz
# fetch -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/doc.txz
# fetch -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/base-dbg.txz
# fetch -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/kernel-dbg.txz
# fetch -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/lib32-dbg.txz
# fetch -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/src.txz
# fetch -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/mports.txz

And here’s the convenience script:

# vi /usr/local/www/pxe/

Put the following into it:

ARCH=`uname -m`
mkdir -p /usr/freebsd-dist
fetch${ARCH}/${RELEASE}-RELEASE/MANIFEST -o /usr/freebsd-dist/MANIFEST

Now you can PXE-boot mfsBSD as prepared in the FreeBSD section. After logging in as root execute the following command line that will take you to the installer:

# fetch && sh

When given the choice to select the installation source make sure to select “Other” and point to the right URL – in my example it would be, then install the OS as you’re used to.

Watch out for some options in the installer, though! MidnightBSD is mostly on par with FreeBSD 11.4. If you enable e.g. newer hardening options that the installer knows but the target OS doesn’t, you might run into trouble (not sure, tough, I didn’t think of this until after my test installation).

What’s next?

I didn’t plan this post to get that long, especially NetBSD surprised me. Eventually I decided to cover HardenedBSD and MidnightBSD as well and accept that this is a BSD-only article. So the next one will add various Linuxen and other operating systems to the mix.

Multi-OS PXE-booting from FreeBSD 12: Required services (pt. 2)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://

The previous post was about what lead me to do this in the first place, featured a little excursion for people new to PXE and most importantly detailed the setup of a FreeBSD router that will be turned into a PXE server in this article.

While I originally intended to show how to boot Ubuntu in this part, things have changed a little. I realized that the software choices I made might not be what a lot of people would have chosen. Therefore I did a little extra work and present my readers with multiple options. This made the article long enough even without the Ubuntu bits which I wanted to cover in part part 3 instead (it was moved to part 4, however).

Current state and scope

Today we will make the machine offer all the services needed for network installations of many Open Source operating systems. My examples are all IPv4, but it should be possible to adapt this to IPv6 fairly easily. As this is *nix, there’s always multiple ways to do things. For software that does not come included in FreeBSD’s base installation, I’ll be giving two options. One will be less commonly deployed but have some advantages in my book. The other will achieve the same goal with a more popular solution.

The machine that I use is an old piece of metal that has 6 NICs – which is why I like to still use it for tinkering with network-related stuff. After part 1, we left off with a gateway that has an Internet-facing adapter em0 which gets its IP via DHCP from my actual router. We’re also using em5 which is statically configured to have the IP and is connected to a separate switch. There’s an unbound nameserver running and serving that interface and a pf firewall is active doing NAT.

This means that everything is in place to serve any host connected to said switch, if it’s manually configured to use a static address in the IP range and with default router and nameserver set to Let’s start by getting rid of that “manually configured” requirement!

Excursion: DHCP basics

We do so by configuring the machine as a DHCP server handing out IP addresses for the subnet on the 6th NIC. DHCP servers work by listening for DHCP requests which are broadcasted on the network (as the client does not have it’s own IP, yet). When receiving one, it will have a look at its configuration: Is there anything special for the host asking? Is the MAC address included with the request maybe blacklisted? Is there a reserved IP to be handed to this specific machine? Or any particular option to send to the “class” of device asking?

In our simple case there aren’t really any bells and whistles involved. So it will have a look at the IP pool that it manages and if it can find an unused one, it will answer with a DHCP offer. The latter is a proposal for an IP to be leased by the client and also includes various network-related information: Usually at least the netmask, router and nameserver. A lot of additional information can be provided in form of options; you can point the client joining the network to a time server if you have one, inform about the domain being used and much, much more (even custom options are possible if you need them).

For PXE booting to work we need to make use of two particular options: We need the PXE code in the firmware to know which server to turn to if it wants to load the NBP (Network Bootstrap Program) from the net. It also needs to know what the file to ask for is called.

DHCP servers

There are multiple DHCP servers out there. If I were doing this on Linux, I’d probably just pick Dnsmasq and be done: As the name implies, it does DNS. But it also does DHCP and TFTP (which we are in need of as well) and supports PXE. But FreeBSD comes with its own TFTP daemon that I’m going to use and I actually prefer the Unix way over all-rounder software: Do one thing and do it well!

The first thing that comes to mind in terms of DHCP servers is ISC’s DHCPd. It’s small, simple to use (at least for our use case), battle-tested and in use pretty much everywhere. It’s old, though, not extremely fast and certainly not as flexible as you might wish for today. This (among other things) lead the ISC to start a new project meant as a replacement: Kea.

The latter is a modern DHCP server with a lot of nice new features: It is a high-performance solution that’s extensible, supports databases as backends, has a web GUI (Stork) available and more. But since DHCPd works well enough, adoption of Kea has been slow. There are a couple of downsides to it, too: First and foremost – its configuration is written in JSON. Yes, JSON! While there are legitimate use cases for that format, configuration is not one of them if you ask me. That was a terrible choice. Kea also pulls in big dependencies like the boost C++ libraries not everybody is fond of.

IMO the benefits of Kea outweight the drawbacks (if it wasn’t for the JSON configuration, I’d even state: clearly). But it’s your choice of course.

DHCP server option 1: Modern-day Kea

Alright, let’s give Kea a try, shall we? First we need to install it and then edit the configuration file:

# pkg install -y kea
# vi /usr/local/etc/kea/kea-dhcp4.conf

The easiest thing to do is to delete the contents and paste the following. Then adapt it to your network and save:

"Dhcp4": {
    "interfaces-config": {
        "interfaces": [ "em5/" ]
    "control-socket": {
        "socket-type": "unix",
        "socket-name": "/tmp/kea4-ctrl-socket"
    "lease-database": {
        "type": "memfile",
        "lfc-interval": 3600
    "expired-leases-processing": {
        "reclaim-timer-wait-time": 10,
        "flush-reclaimed-timer-wait-time": 25,
        "hold-reclaimed-time": 3600,
        "max-reclaim-leases": 100,
        "max-reclaim-time": 250,
        "unwarned-reclaim-cycles": 5

    "renew-timer": 900,
    "rebind-timer": 1800,
    "valid-lifetime": 3600,
    "next-server": "",
    "boot-file-name": "pxelinux.0",

    "option-data": [
            "name": "subnet-mask",
            "data": ""
            "name": "domain-name-servers",
            "data": ""
            "name": "domain-name",
            "data": ""
            "name": "domain-search",
            "data": ""

    "subnet4": [
            "subnet": "",
            "pools": [ { "pool": " -" } ],
            "option-data": [
                    "name": "routers",
                    "data": ""

    "loggers": [
        "name": "kea-dhcp4",
        "output_options": [
                "output": "/var/log/kea-dhcp4.log"

        "severity": "INFO",
        "debuglevel": 0

Yes, looks pretty bad, I know. But that’s only the representation; if something better had been used (say YAML), it’d be about 50 lines instead of 75, be much more readable and above all: less error-prone to edit. Oh well. If you can ignore the terrible representation, the actual data is not so bad and pretty much self-explaining.

I’d like to point you at the “next-server” and “boot-file-name” global options that I set here. These are required for PXE booting by pointing to the server hosting the NBP and telling its file name. Leave them out and you will still have a working DHCP server if you don’t need to do PXE. While this configuration works, you will likely want to extend it for production use.

With the config in place, let’s enable and start the daemon:

# sysrc kea_enable="YES"
# service kea start

A quick look if a daemon successfully bound to port 67 and is listening doesn’t hurt:

# sockstat -4l | grep 67
root     kea-dhcp4  1480  14 udp4         *:*

Ok, there we are. We now have a DHCP service on our internal network!

DHCP server option 2: Venerable ISC DHCPd

So you’d like to play simple and safe? No problem, going with DHCPd is not a bad choice. But first we need to install it and edit the configuration file:

# pkg install -y isc-dhcp44-server
# vi /usr/local/etc/dhcpd.conf

Delete everything. Then add the following (adjust to your network structure, of course!) and save:

option domain-name "";
option domain-name-servers;
option subnet-mask;
default-lease-time 600;
max-lease-time 7200;
ddns-update-style none;
log-facility local7;

filename "pxelinux.0";
subnet netmask {
  option routers;

Mind the “next-server” and “filename” options which define the server to get the NBP from as well as the file name of that. You can leave out that block and will still have a working DHCP server – but it won’t allow for PXE booting in that case. I’d also advice you to do a bit of reading and probably do a more comprehensive configuration of DHCPd.

Next thing to do is to enable DHCPd, confine it to serve requests coming in from one particular NIC only and start the service:

# sysrc dhcpd_enable="YES"
# sysrc dhcpd_ifaces="em5"
# service isc-dhcpd start

Quick check to see if the service is running and binding on port 67:

# sockstat -4l | grep 67
dhcpd    dhcpd      1396  8  udp4   *:67                  *:*

Looking good so far, DHCP should be available on our internal net.

Optional: Checking DHCP

If you want to make sure that your DHCP server is not only running but that it can also be reached and actually does what it’s meant to, you can either just try to power up a host in the network and configure it to get its IP via DHCP. Or you can for example use the versatile nmap tool to test DHCP discovery from any host on that network:

# pkg install -y nmap
# nmap --script broadcast-dhcp-discover
Starting Nmap 7.91 ( ) at 2021-01-24 17:56 CET
Pre-scan script results:
| broadcast-dhcp-discover: 
|   Response 1 of 1: 
|     Interface: em0
|     IP Offered:
|     DHCP Message Type: DHCPOFFER
|     Subnet Mask:
|     Router:
|     Domain Name Server:
|     Domain Name:
|     IP Address Lease Time: 1h00m00s
|     Server Identifier:
|     Renewal Time Value: 15m00s
|_    Rebinding Time Value: 30m00s
WARNING: No targets were specified, so 0 hosts scanned.
Nmap done: 0 IP addresses (0 hosts up) scanned in 10.56 seconds

# pkg delete -y nmap

All right! DHCP server is working and happily handing out leases.


The Trivial File Transfer Protocol daemon is up next. FreeBSD ships with a TFTP daemon in the base system, so we’re going to use that. It will not be used by itself but instead from the inetd super daemon. To enable TFTP, we just need to put one line in the inetd configuration file:

# echo "tftp    dgram   udp     wait    root    /usr/libexec/tftpd      tftpd -l -s /usr/local/tftpboot" >> /etc/inetd.conf

Now we need to create the directory that we just referenced, as well as a subdirectory which we’re going to use and create a file there:

# mkdir -p /usr/local/tftpboot/pxelinux.cfg
# vi /usr/local/tftpboot/pxelinux.cfg/default

Put the following in there (for now) and save:

DEFAULT vesamenu.c32

All is set, let’s enable and start the service now:

# sysrc inetd_enable="YES"
# service inetd start

Again we can check real quick if the service is running:

# sockstat -4l | grep 69
root     inetd      1709  6  udp4   *:69                  *:*

Good, so now we can test to fetch the configuration file from either our server or from any FreeBSD machine in the network:

# tftp
tftp> get pxelinux.cfg/default
Received 292 bytes during 0.0 seconds in 1 blocks
tftp> quit
# rm default

Everything’s fine just as expected.

File Server

The remaining piece we need to set up is a means to efficiently transfer larger files over the wire – i.e. not TFTP! You can do it via FTP and use FreeBSD’s built-in FTP daemon. While this works well, it is not the option that I’d recommend. Why? Because FTP is an old protocol that does not play nicely with firewalls. Sure, it’s possible to do FTP properly, but that’s more complex to do than using something else like a webserver that speaks HTTP.

If you want to go down that path, there are a lot of options. There’s the very popular and feature-rich Apache HTTPd and various more light-weight solutions like LighTTPd and many more. I generally prefer OpenBSD’s HTTPd because it is so easy to work with – and when it comes to security or resisting feature creep its developers really mean it. If I need to do something that it cannot do, I usually resort to the way more advanced (and much more popular) Nginx.

Pick any of the two described here, go with FTPd instead or just ignore the following three sections and set up the webserver that you prefer to deploy.

If you didn’t opt for FTP, as a first step create a directory for the webserver and change the ownership:

# mkdir -p /usr/local/www/pxe
# chown -R www:www /usr/local/www/pxe

File Server option 1: OpenBSD’s HTTPd

Next is installing the program and providing the desired configuration. Edit the file:

# pkg install -y obhttpd
# vi /usr/local/etc/obhttpd.conf

Delete the contents and replace it with the following, then save:

chroot "/usr/local/www"
logdir "/var/log/obhttpd"
server "" {
        listen on port 80
        root "/pxe"
        log style combined

Super simple, eh? That’s part of the beauty of obhttpd. OpenBSD follows the “sane defaults” paradigm. That way you only have to configure stuff that is specific to your task as well as things where you want to change the defaults. Surprisingly, this configuration does it – and there’s really not much I’d change for a production setup if it is the only site on this server.

It’s always a good idea to check if the configuration is valid, so let’s do that:

# obhttpd -nf /usr/local/etc/obhttpd.conf
configuration OK

If you ever need to debug something, you can start the daemon in foreground and more verbosely by running obhttpd -dvv. Right now the server would not start because the configured log directory does not exist. So this would be a chance to give debugging a try.

Let’s create the missing directory and then enable and start the service:

# mkdir /var/log/obhttpd
# sysrc obhttpd_enable="YES"
# service obhttpd start

As always I prefer to take a quick look if the daemon did bind the way I wanted it to:

# sockstat -4l | grep httpd
www      obhttpd    1933  7  tcp4         *:*
www      obhttpd    1932  7  tcp4         *:*
www      obhttpd    1931  7  tcp4         *:*

Looks good.

File Server option 2: Nginx

Next thing is installing Nginx and providing a working configuration:

# pkg install -y nginx
# vi /usr/local/etc/nginx/nginx.conf

Erase the example content and paste in the following:

user  www;
error_log  /var/log/nginx/error.log;
worker_processes  auto;

events {
    worker_connections  1024;

http {
    include       mime.types;
    default_type  application/octet-stream;
    keepalive_timeout  65;

    server {
        listen       80;
        location / {
            root   /usr/local/www/pxe;

This is by no means a perfect configuration but only an example. If you want to deploy Nginx in production, you’ll have to further tune it towards what you want to achieve. But now let’s enable and start the daemon:

# sysrc nginx_enable="YES"
# service nginx start

Time for the usual quick check:

# sockstat -4l | grep nginx
www      nginx      1733  6  tcp4   *:80                  *:*
www      nginx      1732  6  tcp4   *:80                  *:*
root     nginx      1731  6  tcp4   *:80                  *:*

Nginx is running and listening on port 80 as it should be.

File Server option 3: FTPd

FTP for you, eh? Here we go. FreeBSD comes with an ftp group but not such a user by default. Let’s create it:

# pw useradd -n ftp -u 14 -c "FTP user" -d /var/ftp -g ftp -s /usr/sbin/nologin

It’s a convention that public data offered via anonymous FTP is placed in a “pub” directory. We’re going to honor that tradition and create the directory now:

# mkdir -p /var/ftp/pub
# chown ftp:ftp /var/ftp/pub

If you intend to use logging, create an empty initial log file:

# touch /var/log/ftpd

Now we need to enable the FTP service for inetd (the “l” flag enables a transfer log, “r” is for operation in read-only mode, “A” allows for anonymous access and “S” is for enabling the download log):

# echo "ftp     stream  tcp     nowait  root    /usr/libexec/ftpd       ftpd -l -r -A -S" >> /etc/inetd.conf

As we intend to run a service that allows local users to log in via FTP, we need to consider the security implications of this. In my case I have created the “kraileth” user and it has the power to become root via doas. While OpenSSH is configured to only accept key-based logins, FTP is not. The user also has a password set – which means that an attacker who suspects that the user might exist, can try brute-forcing my password.

If you’re the type of person who is re-using passwords for multiple things, take this scenario into consideration. Sure, this is an internal server and all. But I recommend to get into a “security first” mindset and just block the user from FTP access, anyway. To do so, we just need to add it to the /etc/ftpusers file:

# echo "kraileth" >> /etc/ftpusers

Now let’s restart the inetd service (as it’s already running for TFTP) and check it:

# service inetd restart
# sockstat -4l | grep 21
root     inetd      1464  7  tcp4   *:21                  *:*

Ready and serving!

Optional: Checking File Server service

Time for a final test. If you’re using a webserver, do this:

# echo "TestTest" > /usr/local/www/pxe/test
# fetch
test                                             9  B 8450  Bps    00s
# cat test
# rm test /usr/local/www/pxe/test

If you’re using FTP instead:

# echo "TestTest" > /var/ftp/pub/test
# fetch
test                                                     9  B 9792  Bps    00s
# cat test
# rm test /var/ftp/pub/test

Everything’s fine here, so we can move on.

What’s next?

In part 3, we’re finally going to add data and configuration to boot multiple operating systems via PXE.

Multi-OS PXE-booting from FreeBSD 12: Introduction (pt. 1)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://

This is an introductory article; if you’re familiar with PXE you will want to skip the excursion but may be interested in the “Why”. The article ends with the post-installation setup of my test machine, turning it into a simple router so that the next article can start with the actual PXE-related setup.

Also see part 2 here and part 3 here as well as part 4 here.


These days I decided to reactivate an old laptop of mine. It has processor of an older generation (Ivy Bridge) but it’s an i7 – and it has 24 GB of RAM, making it a somewhat nice machine for certain things. Problem with it is: The USB does not work (I think that the controller was fried)!

So how do I get a current OS installed on there as I cannot use the CD emulator I’d normally use? Sure, I could always burn a CD, but I don’t feel like it (unnecessary waste). I could open it up, remove the drive, place it in another machine, install the OS and then put it back. Nah, not too much fun, either. How about doing a network install then?

When I thought of that I realized that I had some old notes from about a year ago somewhere on my workstation. I originally meant to blog about that topic but never actually wrote the post. So here’s how to do what the post’s title says – updated to FreeBSD 12.2 and Ubuntu 20.04!

While I have a clear favorite OS and very much appreciate what FreeBSD has to offer for me as the administrator of a fleet of servers, there really is no reason to turn a blind eye to other Unix-like systems. For one thing it totally makes sense to always pick the best candidate for a job (which might not in all cases be one OS). That and the fact that you cannot judge which one is best suited if you don’t have at least some level of familiarity with various options. But in addition to that my employer runs a heterogeneous environment, anyway – and while I’m mostly occupied with the BSD side of things, there’s simply way, way too many Linux machines around to even think about avoiding them altogether all the time.

Why PXE-boot Linux from FreeBSD?

After an update, the old means of installing Linux servers as we do it at work had stopped working reliably. I looked at it briefly but soon found that too many things weren’t set up like you’d do it today. Therefore I decided that it might make more sense to start fresh. And while at it – wouldn’t it make sense to try and combine the Linux and FreeBSD PXE servers on one machine? That would mean one less server to run after all.

The next installation due was for a customer who requested an Ubuntu server. As a matter of fact we are transitioning to use that distribution more often for our internal infrastructure, too (decision by management, certainly not my choice!). For that reason one weekend I ended up doing something that I hadn’t done in a while: installing a fresh Ubuntu 16.04 system on a test machine I. After doing this I could write a whole post about how bad the installer actually is (static IP addressing, anyone??), but I don’t want this to turn into a rant.

So let’s just mention one single complaint: Installing takes quite a long time (I never understood why Debian and derivatives install stuff that they remove again later during the “cleaning up” phase…). Before the installation was even finished, I admittedly already had mixed feelings about this new system. But then this was what happened on the first boot:

[ TIME ] Timed out waiting for device dev-disk-by\x2duuid-0ef05387\x2d50d9\x2d4cac\x2db96\x2d9808331328af.device.
[DEPEND] Dependency failed for /dev/disk/by-uuid/0ef05387-50d9-4cac-b796-9808331328af.
[DEPEND] Dependency failed for Swap.
[  *** ] A start job is running for udev Coldplug all Devices (3min 34s / no limit)

You’ve got to be kidding! A freshly installed system going sideways like that? Sorry Ubuntu that’s not the kind of stuff that I like wasting my time with! I don’t even care if it’s systemd’s fault or something else. The boss “preferably” wanted an Ubuntu system – I gave it a try and it failed me. Ah, screw it, let’s deploy something I know of that it actually works (same hardware BTW, before anybody wants to blame that for a problem that’s definitely Ubuntu’s fault)!

I had a freshly installed FreeBSD with static IP configuration (which you probably want to use for a DHCP / PXE boot server after all) in a fraction of the time that the Ubuntu installation took. And it goes without saying: System starts as one would expect.

Excursion: PXE – An Overview

While there have been earlier methods for making a machine boot over the network, PXE (Preboot eXecution Environment) is what you want to rely on if you need to do that today. It is a specification widely implemented (in fact just about everywhere) and chances are very low that you will run into problems with it. Have a look around in your computer’s EFI or BIOS, often PXE-booting is disabled (and if you want to use it just once to install an OS on the machine, I recommend that you disable it again afterwards). How to make a machine boot over the net depends on its EFI / BIOS. Figure out how to do it four your metal and give it a try.

On your average home environment not too much should happen. The PXE environment will probe the network for the information that it needs, receive none and eventually give up. But what information does it need and which components are involved?

Well, it needs to load an operating system “from the net” – which means from another machine. To be able to talk to other hosts on the network, it needs to become part of said net. For this it needs to know the network parameters and requires a usable, unique IP address for itself. It can get all of that from a DHCP (Dynamic Host Configuration Protocol) server. If configured correctly, the daemon will accept DHCP requests and provide the client with a suggested IP as well as information about the network (like the netmask, default router, nameservers on the net, etc). It can also tell the client from where it can load the NBP (Network Bootstrap Program).

The latter is a piece of software, usually a special bootloader. It will be downloaded via TFTP (Trivial File Transfer Protocol). Think of it as very similar to FTP but much, much more primitive. The NBP is executed when the transfer is completed. It will then download its configuration file and then act accordingly, most likely fetching an operating system kernel and additional files, then booting the kernel.

TFTP is what the PXE bootcode speaks and uses. But due to its very simplistic nature, TFTP is not well fit for larger file transfers. Therefore such files are usually loaded via other means once the kernel has booted and more options are available. FreeBSD likes to use NFS, while on Linux HTTP is often used to receive a filesystem or installation medium image.

So the following services are involved when setting up a PXE boot server:

  • DHCP server
  • TFTP server
  • Webserver / NFS service stack

Preparing a FreeBSD router

Today I’m redoing my previous work in my home lab with FreeBSD 12.2. The machine that I’m using is pretty old (32-bit only Atom CPU!) but it has 6 Ethernet ports and still works well for network tinkering. I got it for free some years ago, so there’s really no reason to complain.

When it comes to the installation, in this case I’m going with MBR + UFS which I normally wouldn’t do for a amd64 machine. It’s a standard 12.2 installation otherwise with my “kraileth” user (member of the wheel group) added during the installation.

First thing to do is copying my public SSH key onto the server and then SSHing into the machine:

% ssh-copy-id -i ~/.ssh/id_ed25519 kraileth@hel.local
% ssh kraileth@hel.local

Now I switch to the root user, disallow SSH password-based login and restart the SSH daemon:

% su -l
# sed -i "" 's/#ChallengeResponseAuthentication yes/ChallengeResponseAuthentication no/' /etc/ssh/sshd_config
# service sshd restart

I don’t need sendmail on the machine, so off it goes:

# sysrc sendmail_enable="NONE"

Next is bootstrapping the package manager, installing the unbound DNS server and – for convenience – doas (a simpler sudo replacement). Writing a one-line config for doas is fine for my use case:

# env ASSUME_ALWAYS_YES=TRUE pkg bootstrap
# pkg install -y doas unbound
# echo "permit :wheel" > /usr/local/etc/doas.conf

Then unbound needs to be configured. To do so, edit the config file like this:

# vi /usr/local/etc/unbound/unbound.conf

Delete all the content, put the following in there (adjust to your network, of course) and save:

        verbosity: 1
        access-control: allow

Keep in mind that this is not a production configuration! You will have to do some research on the program if you want to put it into production. Do at the very least read the more than thousand lines of comments that you just deleted (you can find it in /usr/local/etc/unbound/unbound.conf.sample). The above config will make the daemon bind on the IP configured via the “interface” statement and allow DNS recursion for any host in the subnet defined via “access-control”. Let’s enable the daemon now (so it will start after rebooting the machine):

# sysrc unbound_enable="YES"

During installation I chose the first NIC (em0) to be configured via DHCP as offered by my home router. This machine will act as a DHCP server for another network, so I assign a static address to the respective interface (em5) that I have connected to a spare switch I had around. It will also act as a gateway between the two networks it is part of:

# sysrc ifconfig_em5="inet netmask"
# sysrc gateway_enable="YES"

To actually provide Internet connectivity to hosts in the subnet, we have to enable NAT (Network Address Translation). Otherwise any host that is contacted via an IP in that range will have no idea how to answer, even though our gateway forwarded the message to it. NATing is done via the firewall. FreeBSD offers three of them. Pf is my absolute favorite, but ipfw is ok, too. I wouldn’t recommend ipf – it’s old and there is really no reason to use it if you’re not already doing so and don’t want to switch.

First we’ll create a configuration file:

# vi /etc/pf.conf

Now put the following in there (again: Adjust to your network) and save:

localnet = $int_if:network

scrub in all
nat on $ext_if from $localnet to any -> ($ext_if)

This defines commonly used three macros: One for the external and internal NIC and one for the local network. It then turns scrubbing on (you almost always want to do this and I suggest that you look up what it does). The final line does the magic of the actual NAT.

You will want to define a more complex firewall configuration if you plan to use this gateway in production. But this is enough for my (demonstration) purposes. I suggest that you follow along and when you made sure that PXE booting works, tighten up security – then test if everything still works. Firewalling is a topic of its own, and in fact not a small one, though. If you’re new to it it makes perfect sense to do some reading. There’s lots of free material on the net as well as a great “Book of Pf” by Peter Hansteen if you prefer.

So let’s enable Pf on startup:

# sysrc pf_enable="YES"
# sysrc pf_rules="/etc/pf.conf"

Reminder: If you want to do things right, you will probably want to enable pflog, too, for example. This post is meant to point you in the right direction and to get stuff to work quickly. Familiarizing yourself with the various components used is your own homework.

It’s always a good idea to update the system to the latest patchlevel:

# freebsd-update fetch install
# freebsd-version -kr

Ok, looks like kernel-related updates were installed, so it’s time to reboot:

# shutdown -r now

That’s it for the preparation work. Once the server comes back up again it should function as a router. If you configure a host with a static address in the network, it should be able to have DNS queries answered as well as reach the Internet.

What’s next?

In part 2 we’re going to complete the setup of the PXE boot server so that a client can boot into Ubuntu Linux. The rest will be in part 3.