Multi-OS PXE-booting from FreeBSD 12: PXE menu and *BSD (pt. 3)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/multi-os_pxe-booting_from_fbsd_pt3.gmi

Post 1 of this mini series is about what lead me to do this in the first place, features a little excursion for people new to PXE and details the setup of a FreeBSD router.
Post 2 discusses setting up the required daemons for DHCP, TFTP and HTTP / FTP. Each component that is not in FreeBSD’s base system is briefly discussed and two options to pick from are covered.

At the end of part 2, the situation is as follows: A client in the 10.11.12.0/24 subnet attempting to PXE boot will get an IP address via DHCP and be told where to get the NPB (Network Bootstrap Program). It will then attempt to fetch it via TFTP. There’s just one problem: That is not there, yet! We’ll fix that in a minute. When we’re done here, you’ll have a fully working PXE server that offers multiple BSD operating systems to boot into (Linux and more are covered in part 4). This article focuses on BIOS (“legacy”) booting; if you want to boot EFI-only machines you’ll have to adapt the configuration examples given here to that. I also assume using HTTP here – if you opted for FTP you will have to adapt the commands used in the examples.

Network Bootstrap Program

There are a lot of NBPs available. Usually each operating system has its own which is tuned towards its specific duty: FreeBSD has one, OpenBSD has another and Linux has several. These are not ordinary programs; they need to cope with a very resource-constrained environment and cannot depend on any external libraries. While writing boot code is challenging enough, adding network booting capabilities doesn’t make things any easier. This is why most NBPs are as simple as possible.

As a result of that, the NBPs usually know how to boot exactly one operating system. Since we want to set up a multi-OS PXE server this is quite unfortunate for our use case. There are two ways to work around this problem:

  1. Provide various NBPs and use DHCP to distinguish between clients
  2. Use an NBP that supports a menu to select which one to boot next

As usual there’s pros and cons to both. Letting DHCP do the magic requires a much more complex DHCP configuration. It’s also much less flexible. The boot menu approach is simple and flexible, but more complicated if you are also interested in automation. I do like automation, but I decided in favor of using a boot menu for this article because it’s easier to follow. It is also a good achievement to build upon once you’re comfortable with DHCP and feel ready for advanced settings.

It is possible to use one NBP to fetch and execute another one. This process is known as chainloading. For some operating systems that is the best choice to add them to a menu system.

There’s three popular options that we have for an NBP which fits our needs:

1. GRUB
2. PXELINUX (from Syslinux) and
3. iPXE

GRUB and I never made friends. I used it for a while after switching from LILO only to ditch it for Syslinux when I learned of that. I have to use it on many systems, but when I have a choice, I choose something else. The iPXE project is very interesting. It’s the most advanced (but also most involved) of the three options. If you’re curious about just how far you can take PXE booting, at least have a look at it. For this article, we’ll go with PXELINUX.

Pxelinux

Pxelinux is available via packages on FreeBSD. It does pull in some dependencies that I don’t want on my system however. For that reason we’re going to fetch the package instead of installing it. Then we extract it manually:

# pkg fetch -y syslinux
# mkdir /tmp/syslinux
# tar -C /tmp/syslinux -xvf /var/cache/pkg/syslinux-6.03.txz

Now we can cherry-pick the required files:

# cp /tmp/syslinux/usr/local/share/syslinux/bios/core/lpxelinux.0 /usr/local/tftpboot/pxelinux.0
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/elflink/ldlinux/ldlinux.c32  /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/menu/vesamenu.c32 /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/lib/libcom32.c32 /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/libutil/libutil.c32 /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/modules/pxechn.c32 /usr/local/tftpboot/
# cp /tmp/syslinux/usr/local/share/syslinux/bios/memdisk/memdisk /usr/local/tftpboot/
# rm -r /tmp/syslinux

The first one is the modular NBP itself. It requires some modules – the c32 files. The pxechn and memdisk modules are optional – they are required for some of the operating systems examples here but not all. You can leave them out if you don’t need them. Restart the inetd service now and you will be able to PXE boot to the menu:

# service inetd restart

Keep in mind: Restart the inetd service whenever you added a file to or edited any in the tftpboot directory!

Tip 1: You can use make use of gzipped files as Pxelinux supports that. This way you can decrease loading times by serving smaller images over the net.

Tip 2: I’m using gzip in my examples but if you really want to fight for the last byte, use zopfli instead. It’s a compression program that produces gzip-compatible output but takes much more processor time to create optimized archives. As decompression time is unaffected it’s a price you have to pay only once. Consider using it if you want the best results.

Submenus

Pxelinux is hard-coded to load pxelinux.cfg/default via TFTP and use that as its configuration. If you plan to only use few of the OS examples shown here, that config is sufficient as you can put everything into there. Once you feel that your menu is becoming overly crowded, you can turn it into a main menu and make use of submenus to group things as I do it here. This works by putting the various OS entries in different config files.

If you don’t want submenus, skip the next step and put all the menu entries that go into something other than pxelinux.cfg/default in my examples right into that file instead – and leave out the reference back to the main menu since they don’t make any sense if you’re using a flat menu anyway.

In the previous post we already created the file /usr/local/tftpboot/pxelinux.cfg/default. Append the following to it to create a submenu for the BSDs we’re covering today:

MENU TITLE PXE Boot Menu (Main)

LABEL bsd-oses
        MENU LABEL BSD Operating Systems
        KERNEL vesamenu.c32
        APPEND pxelinux.cfg/bsd

Now create the file referenced there:

# vi /usr/local/tftpboot/pxelinux.cfg/bsd

and put the following text in there:

MENU TITLE PXE Boot Menu (BSD)

LABEL main-menu
        MENU LABEL Main Menu
        KERNEL vesamenu.c32
        APPEND pxelinux.cfg/default

Alright, preparation work is done, let’s finally add some operating system data!

FreeBSD 12.2

PXE booting FreeBSD is actually not such an easy thing to do if you want to avoid using NFS shares. Fortunately there is mfsBSD, a project that provides tools as well as releases of special FreeBSD versions that can be booted over the net easily. We’re going to use that.

There are multiple variants for download: The standard one, the “special edition” and a “mini edition”. The special edition comes with the distribution tarballs on the ISO – you may want to use that one for installation purposes. If you just want a FreeBSD live system (e.g. for maintenance and repairs) or use you owr mirror (see below), the standard edition is for you since it is much smaller and thus boots way faster.

Let’s make the image available via HTTP:

# mkdir -p /usr/local/www/pxe/bsd/fbsd
# fetch https://mfsbsd.vx.sk/files/iso/12/amd64/mfsbsd-12.2-RELEASE-amd64.iso -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso
# gzip -9 /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso

Now edit the pxelinux.cfg/bsd file and append:

LABEL fbsd-pxe-install
        MENU LABEL Install FreeBSD 12.2 (PXE)
        MENU DEFAULT
        KERNEL memdisk
        INITRD http://10.11.12.1/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso
        APPEND iso raw

That’s it. You can now PXE-boot into FreeBSD.

Installation using mfsBSD

Login with user root and password mfsroot. It includes a “zfsinstall” script that you may want to take a look at. There’s a lot more to mfsBSD, though. Its tools allow you to easily roll your own customized images. If a way to include packages or files, use a custom-built kernel and things like that sounds like something that would be useful for you, take a closer look. I cannot go into more detail here – it’s a topic of its own and would deserve an entire article dedicated to it. In case you just want to use the familiar FreeBSD installer bsdinstall, read on.

Mirroring distfiles and fixing bsdinstall

If you want to install FreeBSD over PXE more than once, it makes sense to provide a local distfile mirror. Since we have a fileserver running anyway, there’s really nothing more to it than getting the distfiles and putting them into the right place. At the very minimum get the following three files:

# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/MANIFEST -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/MANIFEST
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/base.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/base.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/kernel.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/kernel.txz

Depending on which distfiles you usually install, also get any or all of the following files:

# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/base-dbg.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/base-dbg.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/kernel-dbg.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/kernel-dbg.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/lib32-dbg.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/lib32-dbg.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/ports.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/ports.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/src.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/src.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/tests.txz -o /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/tests.txz

At this point in time, bsdinstall is broken on mfsBSD. The reason is that the distfile manifest is missing. I think about getting this fixed upstream, so in the future try and see if the following part is obsolete before using it. But for now, let’s create a simple shell script in the webroot directory for convenience:

# vi /usr/local/www/pxe/fbsd.sh

Put the following into the script:

#!/bin/sh
ARCH=`uname -m`
RELEASE=`uname -r | cut -d"-" -f1`
mkdir -p /usr/freebsd-dist
fetch http://10.11.12.1/bsd/fbsd/${ARCH}/${RELEASE}-RELEASE/MANIFEST -o /usr/freebsd-dist/MANIFEST
bsdinstall

Now if you PXE-booted mfsBSD and logged in as root, you just need to execute the following command line and will then be able to use the installer as you are used to it:

# fetch http://10.11.12.1/fbsd.sh && sh fbsd.sh

When you are to select the installation source, there is an “Other” button at the bottom of the window. Choose that and point to your distfile mirror – in my example http://10.11.12.1/bsd/fbsd/amd64/12.2-RELEASE. Happy installing!

One more hint: You may want to look into the environment variables that bsdinstall(8) provides. I briefly attempted to automatically set the URL to the distfile mirror but couldn’t get it working. As I was already running out of time with this article I haven’t looked deeper into it. If anybody figures it out I’d appreciate sharing your solution here.

OpenBSD 6.8

Adding OpenBSD as an option is trivial. The project provides a ramdisk kernel used for installing the system and a NBP capable of loading it. Let’s get those two files in place – and while the ramdisk kernel is fairly small already, we can chop a couple of bytes off by compressing it:

# mkdir -p /usr/local/tftpboot/bsd/obsd
# fetch https://cdn.openbsd.org/pub/OpenBSD/6.8/amd64/pxeboot -o /usr/local/tftpboot/bsd/obsd/pxeboot
# fetch https://cdn.openbsd.org/pub/OpenBSD/6.8/amd64/bsd.rd -o /usr/local/tftpboot/bsd/obsd/6.8-amd64.rd
# gzip -9 /usr/local/tftpboot/bsd/obsd/6.8-amd64.rd

Now we only need to add the required lines to pxelinux.cfg/bsd:

LABEL obsd-pxe-install
        MENU LABEL Install OpenBSD 6.8 (PXE)
        KERNEL pxechn.c32
        APPEND bsd/obsd/pxeboot

That’s it, the OpenBSD loader can be booted! Since we don’t have the kernel in the assumed default location (“/bsd”) we’d need to tell the loader to “boot bsd/obsd/6.8-amd64.rd.gz”. The loader supports a configuration file, though. So for a little extra convenience we can make it pick up the kernel automatically like this:

# mkdir -p /usr/local/tftpboot/etc
# ln -s /usr/local/tftpboot/etc/boot.conf /usr/local/tftpboot/bsd/obsd/boot.conf
# echo "boot bsd/obsd/6.8-amd64.rd.gz" > /usr/local/tftpboot/bsd/obsd/boot.conf
# echo "# OpenBSD boot configuration" >> /usr/local/tftpboot/bsd/obsd/boot.conf

The pxeboot program comes with the configuraton file name of etc/boot.conf hard-coded. To keep things a little cleaner in the hierarchy that I use, I chose to set a symlink in the obsd directory for reference purposes. And that’s all.

NetBSD 9.1

Let’s add NetBSD! It’s somewhat similar to OpenBSD – but a bit more involved unfortunately. The reason is that the NBP by default does not support a configuration file. It has the ability to use one, but that needs to be activated first. Which is fair enough since it’s only a single command – on NetBSD that is! But let’s worry about this in a minute and first get the NBP as well as the install kernel:

# mkdir -p /usr/local/tftpboot/bsd/nbsd
# fetch https://cdn.netbsd.org/pub/NetBSD/NetBSD-9.1/amd64/installation/misc/pxeboot_ia32.bin -o /usr/local/tftpboot/bsd/nbsd/pxeboot_ia32.bin
# fetch https://cdn.netbsd.org/pub/NetBSD/NetBSD-9.1/amd64/binary/kernel/netbsd-INSTALL.gz -o /usr/local/tftpboot/bsd/nbsd/netbsd-INSTALL.gz

Now we need to add the boot menu entry by adding the following lines to pxelinux.cfg/bsd:

LABEL nbsd-pxe-install
        MENU LABEL Install NetBSD 9.1 (PXE)
        KERNEL pxechn.c32
        APPEND bsd/nbsd/pxeboot_ia32.bin

This is sufficient to load and execute the NetBSD loader. That will then complain that it cannot find the kernel and no hints about NFS were given. Now we have three options:

  1. Manually point the loader to the correct kernel each time
  2. Give the required hint via DHCP
  3. Try to enable the loader configuration

Typing in “tftp:bsd/nbsd/netbsd-INSTALL.gz” is probably fair enough if you are doing NetBSD installs very rarely but it gets old quickly. So let’s try out option two!

Modifying DHCP config for NetBSD

The DHCP server needs to be configured to pass a different Boot File name option when answering the NetBSD loader than otherwise. This is done by matching class information. This topic is beyond the scope of this article, so if you are interested, do some reading on your own. I won’t leave you hanging, though, if you just need to get things working.

Here’s what you have to add to the configuration if you’re using Kea – for example right before the “loggers” section:

    "client-classes": [
        {
            "name": "NetBSDloader",
            "test": "option[vendor-class-identifier].text == 'NetBSD:i386:libsa'",
            "boot-file-name": "tftp:bsd/nbsd/netbsd-INSTALL.gz"
        }
    ],

And here the same thing if you are using DHCPd:

if substring (option vendor-class-identifier, 0, 17) = "NetBSD:i386:libsa" {
    if filename = "netbsd" {
        filename "tftp:bsd/nbsd/netbsd-INSTALL.gz";
    }
}

Restart your DHCP server and you should be good to go.

After accomplishing the easy way via DHCP, I also went down to explore the boot.cfg road but ultimately failed. I’m documenting it here anyway in case somebody wants to pick up where I decided to leave it be.

Enabling boot.cfg in the loader

To mitigate the risk of polluting my main system by doing something stupid I chose to do all of this using my unprivileged user. The first thing I did, was fetching and extracting the basic NetBSD 9.1 sources:

% mkdir -p netbsd-9.1 && cd netbsd-9.1
% fetch ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-9.1/source/sets/src.tgz
% tar xvzf src.tgz

The sources for the installboot program we’re looking for are in usr/src/usr.sbin/installboot. I tried to get that thing to build by pointing the compiler at additional include directories and editing quite some header files, hoping to resolve the conflicts with FreeBSD’s system headers and problems like that. It can probably be done but that would take a C programmer – which I am not.

Fortunately NetBSD is the portability star among the BSDs and should be buildable on many other systems. I’ve never done this before but here was the chance. So I installed CVS and checked out the rest of the NetBSD src module:

% doas pkg install -y cvs
% cd netbsd-9.1/usr
% cvs -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -r netbsd-9-1-RELEASE -P src

When exploring the source tree, I found a build script that obviously does all the magic required here. Be warned however that this builds a full cross-toolchain including the complete GCC compiler! Then it builds the “tools” subset of the NetBSD code (which includes the installboot that we’re looking for). On my old and slow Atom-based system this process took 6 hours:

% cd src
% ./build.sh -U -m amd64 -T ~/nbsd tools
===> build.sh command:    ./build.sh -U -m amd64 -T /home/kraileth/nbsd tools
===> build.sh started:    Thu Feb  3 00:13:41 CET 2021
===> NetBSD version:      9.1
===> MACHINE:             amd64
===> MACHINE_ARCH:        amd64
===> Build platform:      FreeBSD 12.2-RELEASE-p1 amd64
===> HOST_SH:             /bin/sh
===> No $TOOLDIR/bin/nbmake, needs building.
===> Bootstrapping nbmake
checking for sh... /bin/sh
checking for gcc... cc

[...]

install ===> config
#   install  /home/kraileth/nbsd/bin/nbconfig
mkdir -p /home/kraileth/nbsd/bin
/home/kraileth/nbsd/bin/amd64--netbsdelf-install -c  -r -m 555 config /home/kraileth/nbsd/bin/nbconfig
===> Tools built to /home/kraileth/nbsd
===> build.sh ended:      Thu Feb  3 6:26:25 CET 2021
===> Summary of results:
         build.sh command:    ./build.sh -U -m amd64 -T /home/kraileth/nbsd tools
         build.sh started:    Thu Feb  4 07:13:41 CET 2021
         NetBSD version:      9.1
         MACHINE:             amd64
         MACHINE_ARCH:        amd64
         Build platform:      FreeBSD 12.2-RELEASE-p1 amd64
         HOST_SH:             /bin/sh
         No $TOOLDIR/bin/nbmake, needs building.
         Bootstrapping nbmake
         MAKECONF file:       /etc/mk.conf (File not found)
         TOOLDIR path:        /home/kraileth/nbsd
         DESTDIR path:        /usr/home/kraileth/netbsd-9.1/usr/src/obj/destdir.amd64
         RELEASEDIR path:     /usr/home/kraileth/netbsd-9.1/usr/src/obj/releasedir
         Created /home/kraileth/nbsd/bin/nbmake
         Updated makewrapper: /home/kraileth/nbsd/bin/nbmake-amd64
         Tools built to /home/kraileth/nbsd
         build.sh ended:      Thu Feb  4 12:26:25 CET 2021
===> .

The “-U” flag enables some trickery to build as an unprivileged user. With “-m” you specify the target architecture (I did use i386 but modified the above lines as that will be what most people will want to use instead!). Finally the “-T” switch allows to specify the installation target directory and the “tools” is the make target to use.

When it was done, I did the following (as root):

# cp /usr/local/tftpboot/bsd/nbsd/pxeboot_ia32.bin /usr/local/tftpboot/bsd/nbsd/pxeboot_ia32.bin.bak
# /usr/home/kraileth/netbsd-9.1/usr/src/tools/installboot/obj/installboot -eo bootconf /usr/local/tftpboot/bsd/nbsd/pxeboot_ia32.bin

This should enable the boot config file on the pxeboot loader. It does change the file and probably even makes the right change. I tried to enable module support via installboot, too and that obviously worked (the NFS module was loaded the next time I chainloaded the NetBSD loader). But for some reason I could not get boot.cfg to do what I wanted. Probably I don’t understand the file properly…

While it’s a bit disappointing to stop so close to the goal, messing with NetBSD so much already took much more time away from the other BSDs than I had imagined. And since I could at least offer a working way this was when I decided to move on.

DragonFly BSD

I attempted to get DragonFly BSD to work but failed. I briefly tried out a setup that includes NFS shares but it didn’t work completely either: Kernel booted but failed to execute /sbin/init for some reason or another. Also I don’t really want to cover NFS in this series – there’s enough material in here already. And without NFS… Well, DragonFly BSD has the same problem that FreeBSD has: It will boot the kernel but then be unable to mount the root filesystem.

While I guess that the mfsBSD approach could likely work for DF, too, this is something much more involved than reasonable for our topic here. I would really like to cover DragonFly here, too, but that’s simply a bit too much. If anybody knows how to get it working – please share your knowledge!

HardenedBSD 12-STABLE

HardenedBSD being a FreeBSD fork, inherited the same characteristics as vanilla FreeBSD. Which means that PXE booting the standard images is not an easy thing to do. HardenedBSD uses the same installer, bsdinstall however and for that reason it’s possible to install HardenedBSD by using mfsBSD as prepared above in the FreeBSD section. We only need to point the installer to a different distfile mirror. Let’s create that one now:

# mkdir -p /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE
# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/MANIFEST -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/MANIFEST
# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/base.txz -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/base.txz
# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/kernel.txz -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/kernel.txz

As with FreeBSD, there are some optional distfiles you may or may not want to mirror, too. Provide what you need for your installations:

# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/base-dbg.txz -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/base-dbg.txz
# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/kernel-dbg.txz -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/kernel-dbg.txz
# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/src.txz -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/src.txz
# fetch https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/tests.txz -o /usr/local/www/pxe/bsd/hbsd/amd64/12-STABLE/tests.txz

Now we’re creating a convenience script for HardenedBSD:

# vi /usr/local/www/pxe/hbsd.sh

Put the following into the script:

#!/bin/sh
ARCH=`uname -m`
MAJOR=`uname -r | cut -d"." -f1`
mkdir -p /usr/freebsd-dist
fetch http://10.11.12.1/bsd/hbsd/${ARCH}/${MAJOR}-STABLE/MANIFEST -o /usr/freebsd-dist/MANIFEST
bsdinstall

Now fire up mfsBSD, login as root and simply run the following command line to start the installer:

# fetch http://10.11.12.1/hbsd.sh && sh hbsd.sh

Select the “Other” button when asked for the installation source. Choose that and point to your distfile mirror – in my example http://10.11.12.1/bsd/hbsd/amd64/12-STABLE. And that’s it.

MidnightBSD 2.0

Since MidnightBSD is a FreeBSD fork as well, it also suffers from the well-known problems related to PXE booting. Again mfsBSD comes to the rescue. Let’s create the distfile mirror first:

# mkdir -p /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/MANIFEST -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/MANIFEST
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/base.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/base.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/kernel.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/kernel.txz

Pick any or all of the remaining optional distfiles to be mirrored, too, if you need them:

# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/lib32.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/lib32.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/doc.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/doc.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/base-dbg.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/base-dbg.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/kernel-dbg.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/kernel-dbg.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/lib32-dbg.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/lib32-dbg.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/src.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/src.txz
# fetch https://discovery.midnightbsd.org/releases/amd64/2.0.3/mports.txz -o /usr/local/www/pxe/bsd/mbsd/amd64/2.0-RELEASE/mports.txz

And here’s the convenience script:

# vi /usr/local/www/pxe/mbsd.sh

Put the following into it:

#!/bin/sh
ARCH=`uname -m`
RELEASE="2.0"
mkdir -p /usr/freebsd-dist
fetch http://10.11.12.1/bsd/mbsd/${ARCH}/${RELEASE}-RELEASE/MANIFEST -o /usr/freebsd-dist/MANIFEST
bsdinstall

Now you can PXE-boot mfsBSD as prepared in the FreeBSD section. After logging in as root execute the following command line that will take you to the installer:

# fetch http://10.11.12.1/mbsd.sh && sh mbsd.sh

When given the choice to select the installation source make sure to select “Other” and point to the right URL – in my example it would be http://10.11.12.1/bsd/mbsd/amd64/2.0-RELEASE, then install the OS as you’re used to.

Watch out for some options in the installer, though! MidnightBSD is mostly on par with FreeBSD 11.4. If you enable e.g. newer hardening options that the installer knows but the target OS doesn’t, you might run into trouble (not sure, tough, I didn’t think of this until after my test installation).

What’s next?

I didn’t plan this post to get that long, especially NetBSD surprised me. Eventually I decided to cover HardenedBSD and MidnightBSD as well and accept that this is a BSD-only article. So the next one will add various Linuxen and other operating systems to the mix.

Multi-OS PXE-booting from FreeBSD 12: Required services (pt. 2)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/multi-os_pxe-booting_from_fbsd_pt2.gmi

The previous post was about what lead me to do this in the first place, featured a little excursion for people new to PXE and most importantly detailed the setup of a FreeBSD router that will be turned into a PXE server in this article.

While I originally intended to show how to boot Ubuntu in this part, things have changed a little. I realized that the software choices I made might not be what a lot of people would have chosen. Therefore I did a little extra work and present my readers with multiple options. This made the article long enough even without the Ubuntu bits which I wanted to cover in part part 3 instead (it was moved to part 4, however).

Current state and scope

Today we will make the machine offer all the services needed for network installations of many Open Source operating systems. My examples are all IPv4, but it should be possible to adapt this to IPv6 fairly easily. As this is *nix, there’s always multiple ways to do things. For software that does not come included in FreeBSD’s base installation, I’ll be giving two options. One will be less commonly deployed but have some advantages in my book. The other will achieve the same goal with a more popular solution.

The machine that I use is an old piece of metal that has 6 NICs – which is why I like to still use it for tinkering with network-related stuff. After part 1, we left off with a gateway that has an Internet-facing adapter em0 which gets its IP via DHCP from my actual router. We’re also using em5 which is statically configured to have the IP 10.11.12.1 and is connected to a separate switch. There’s an unbound nameserver running and serving that interface and a pf firewall is active doing NAT.

This means that everything is in place to serve any host connected to said switch, if it’s manually configured to use a static address in the 10.11.12.0/24 IP range and with default router and nameserver set to 10.11.12.1. Let’s start by getting rid of that “manually configured” requirement!

Excursion: DHCP basics

We do so by configuring the machine as a DHCP server handing out IP addresses for the 10.11.12.0/24 subnet on the 6th NIC. DHCP servers work by listening for DHCP requests which are broadcasted on the network (as the client does not have it’s own IP, yet). When receiving one, it will have a look at its configuration: Is there anything special for the host asking? Is the MAC address included with the request maybe blacklisted? Is there a reserved IP to be handed to this specific machine? Or any particular option to send to the “class” of device asking?

In our simple case there aren’t really any bells and whistles involved. So it will have a look at the IP pool that it manages and if it can find an unused one, it will answer with a DHCP offer. The latter is a proposal for an IP to be leased by the client and also includes various network-related information: Usually at least the netmask, router and nameserver. A lot of additional information can be provided in form of options; you can point the client joining the network to a time server if you have one, inform about the domain being used and much, much more (even custom options are possible if you need them).

For PXE booting to work we need to make use of two particular options: We need the PXE code in the firmware to know which server to turn to if it wants to load the NBP (Network Bootstrap Program) from the net. It also needs to know what the file to ask for is called.

DHCP servers

There are multiple DHCP servers out there. If I were doing this on Linux, I’d probably just pick Dnsmasq and be done: As the name implies, it does DNS. But it also does DHCP and TFTP (which we are in need of as well) and supports PXE. But FreeBSD comes with its own TFTP daemon that I’m going to use and I actually prefer the Unix way over all-rounder software: Do one thing and do it well!

The first thing that comes to mind in terms of DHCP servers is ISC’s DHCPd. It’s small, simple to use (at least for our use case), battle-tested and in use pretty much everywhere. It’s old, though, not extremely fast and certainly not as flexible as you might wish for today. This (among other things) lead the ISC to start a new project meant as a replacement: Kea.

The latter is a modern DHCP server with a lot of nice new features: It is a high-performance solution that’s extensible, supports databases as backends, has a web GUI (Stork) available and more. But since DHCPd works well enough, adoption of Kea has been slow. There are a couple of downsides to it, too: First and foremost – its configuration is written in JSON. Yes, JSON! While there are legitimate use cases for that format, configuration is not one of them if you ask me. That was a terrible choice. Kea also pulls in big dependencies like the boost C++ libraries not everybody is fond of.

IMO the benefits of Kea outweight the drawbacks (if it wasn’t for the JSON configuration, I’d even state: clearly). But it’s your choice of course.

DHCP server option 1: Modern-day Kea

Alright, let’s give Kea a try, shall we? First we need to install it and then edit the configuration file:

# pkg install -y kea
# vi /usr/local/etc/kea/kea-dhcp4.conf

The easiest thing to do is to delete the contents and paste the following. Then adapt it to your network and save:

{
"Dhcp4": {
    "interfaces-config": {
        "interfaces": [ "em5/10.11.12.1" ]
    },
    "control-socket": {
        "socket-type": "unix",
        "socket-name": "/tmp/kea4-ctrl-socket"
    },
    "lease-database": {
        "type": "memfile",
        "lfc-interval": 3600
    },
    "expired-leases-processing": {
        "reclaim-timer-wait-time": 10,
        "flush-reclaimed-timer-wait-time": 25,
        "hold-reclaimed-time": 3600,
        "max-reclaim-leases": 100,
        "max-reclaim-time": 250,
        "unwarned-reclaim-cycles": 5
    },

    "renew-timer": 900,
    "rebind-timer": 1800,
    "valid-lifetime": 3600,
    "next-server": "10.11.12.1",
    "boot-file-name": "pxelinux.0",

    "option-data": [
        {
            "name": "subnet-mask",
            "data": "255.255.255.0"
        },
        {
            "name": "domain-name-servers",
            "data": "10.11.12.1"
        },
        {
            "name": "domain-name",
            "data": "example.org"
        },
        {
            "name": "domain-search",
            "data": "example.org"
        }
    ],

    "subnet4": [
        {
            "subnet": "10.11.12.0/24",
            "pools": [ { "pool": "10.11.12.20 - 10.11.12.30" } ],
            "option-data": [
                {
                    "name": "routers",
                    "data": "10.11.12.1"
                }
            ]
        }
    ],

    "loggers": [
    {
        "name": "kea-dhcp4",
        "output_options": [
            {
                "output": "/var/log/kea-dhcp4.log"

            }
        ],
        "severity": "INFO",
        "debuglevel": 0
    }
  ]
}
}

Yes, looks pretty bad, I know. But that’s only the representation; if something better had been used (say YAML), it’d be about 50 lines instead of 75, be much more readable and above all: less error-prone to edit. Oh well. If you can ignore the terrible representation, the actual data is not so bad and pretty much self-explaining.

I’d like to point you at the “next-server” and “boot-file-name” global options that I set here. These are required for PXE booting by pointing to the server hosting the NBP and telling its file name. Leave them out and you will still have a working DHCP server if you don’t need to do PXE. While this configuration works, you will likely want to extend it for production use.

With the config in place, let’s enable and start the daemon:

# sysrc kea_enable="YES"
# service kea start

A quick look if a daemon successfully bound to port 67 and is listening doesn’t hurt:

# sockstat -4l | grep 67
root     kea-dhcp4  1480  14 udp4   10.11.12.1:67         *:*

Ok, there we are. We now have a DHCP service on our internal network!

DHCP server option 2: Venerable ISC DHCPd

So you’d like to play simple and safe? No problem, going with DHCPd is not a bad choice. But first we need to install it and edit the configuration file:

# pkg install -y isc-dhcp44-server
# vi /usr/local/etc/dhcpd.conf

Delete everything. Then add the following (adjust to your network structure, of course!) and save:

option domain-name "example.org";
option domain-name-servers 10.11.12.1;
option subnet-mask 255.255.255.0;
 
default-lease-time 600;
max-lease-time 7200;
ddns-update-style none;
log-facility local7;

next-server 10.11.12.1;
filename "pxelinux.0";
 
subnet 10.11.12.0 netmask 255.255.255.0 {
  range 10.11.12.10 10.11.12.20;
  option routers 10.11.12.1;
}

Mind the “next-server” and “filename” options which define the server to get the NBP from as well as the file name of that. You can leave out that block and will still have a working DHCP server – but it won’t allow for PXE booting in that case. I’d also advice you to do a bit of reading and probably do a more comprehensive configuration of DHCPd.

Next thing to do is to enable DHCPd, confine it to serve requests coming in from one particular NIC only and start the service:

# sysrc dhcpd_enable="YES"
# sysrc dhcpd_ifaces="em5"
# service isc-dhcpd start

Quick check to see if the service is running and binding on port 67:

# sockstat -4l | grep 67
dhcpd    dhcpd      1396  8  udp4   *:67                  *:*

Looking good so far, DHCP should be available on our internal net.

Optional: Checking DHCP

If you want to make sure that your DHCP server is not only running but that it can also be reached and actually does what it’s meant to, you can either just try to power up a host in the 10.11.12.0/24 network and configure it to get its IP via DHCP. Or you can for example use the versatile nmap tool to test DHCP discovery from any host on that network:

# pkg install -y nmap
# nmap --script broadcast-dhcp-discover
Starting Nmap 7.91 ( https://nmap.org ) at 2021-01-24 17:56 CET
Pre-scan script results:
| broadcast-dhcp-discover: 
|   Response 1 of 1: 
|     Interface: em0
|     IP Offered: 10.11.12.21
|     DHCP Message Type: DHCPOFFER
|     Subnet Mask: 255.255.255.0
|     Router: 10.11.12.1
|     Domain Name Server: 10.11.12.1
|     Domain Name: example.org
|     IP Address Lease Time: 1h00m00s
|     Server Identifier: 10.11.12.1
|     Renewal Time Value: 15m00s
|_    Rebinding Time Value: 30m00s
WARNING: No targets were specified, so 0 hosts scanned.
Nmap done: 0 IP addresses (0 hosts up) scanned in 10.56 seconds

# pkg delete -y nmap

All right! DHCP server is working and happily handing out leases.

TFTP

The Trivial File Transfer Protocol daemon is up next. FreeBSD ships with a TFTP daemon in the base system, so we’re going to use that. It will not be used by itself but instead from the inetd super daemon. To enable TFTP, we just need to put one line in the inetd configuration file:

# echo "tftp    dgram   udp     wait    root    /usr/libexec/tftpd      tftpd -l -s /usr/local/tftpboot" >> /etc/inetd.conf

Now we need to create the directory that we just referenced, as well as a subdirectory which we’re going to use and create a file there:

# mkdir -p /usr/local/tftpboot/pxelinux.cfg
# vi /usr/local/tftpboot/pxelinux.cfg/default

Put the following in there (for now) and save:

DEFAULT vesamenu.c32
PROMPT 0

All is set, let’s enable and start the service now:

# sysrc inetd_enable="YES"
# service inetd start

Again we can check real quick if the service is running:

# sockstat -4l | grep 69
root     inetd      1709  6  udp4   *:69                  *:*

Good, so now we can test to fetch the configuration file from either our server or from any FreeBSD machine in the 10.11.12.0/24 network:

# tftp 10.11.12.1
tftp> get pxelinux.cfg/default
Received 292 bytes during 0.0 seconds in 1 blocks
tftp> quit
# rm default

Everything’s fine just as expected.

File Server

The remaining piece we need to set up is a means to efficiently transfer larger files over the wire – i.e. not TFTP! You can do it via FTP and use FreeBSD’s built-in FTP daemon. While this works well, it is not the option that I’d recommend. Why? Because FTP is an old protocol that does not play nicely with firewalls. Sure, it’s possible to do FTP properly, but that’s more complex to do than using something else like a webserver that speaks HTTP.

If you want to go down that path, there are a lot of options. There’s the very popular and feature-rich Apache HTTPd and various more light-weight solutions like LighTTPd and many more. I generally prefer OpenBSD’s HTTPd because it is so easy to work with – and when it comes to security or resisting feature creep its developers really mean it. If I need to do something that it cannot do, I usually resort to the way more advanced (and much more popular) Nginx.

Pick any of the two described here, go with FTPd instead or just ignore the following three sections and set up the webserver that you prefer to deploy.

If you didn’t opt for FTP, as a first step create a directory for the webserver and change the ownership:

# mkdir -p /usr/local/www/pxe
# chown -R www:www /usr/local/www/pxe

File Server option 1: OpenBSD’s HTTPd

Next is installing the program and providing the desired configuration. Edit the file:

# pkg install -y obhttpd
# vi /usr/local/etc/obhttpd.conf

Delete the contents and replace it with the following, then save:

chroot "/usr/local/www"
logdir "/var/log/obhttpd"
 
server "pixie.example.org" {
        listen on 10.11.12.1 port 80
        root "/pxe"
        log style combined
}

Super simple, eh? That’s part of the beauty of obhttpd. OpenBSD follows the “sane defaults” paradigm. That way you only have to configure stuff that is specific to your task as well as things where you want to change the defaults. Surprisingly, this configuration does it – and there’s really not much I’d change for a production setup if it is the only site on this server.

It’s always a good idea to check if the configuration is valid, so let’s do that:

# obhttpd -nf /usr/local/etc/obhttpd.conf
configuration OK

If you ever need to debug something, you can start the daemon in foreground and more verbosely by running obhttpd -dvv. Right now the server would not start because the configured log directory does not exist. So this would be a chance to give debugging a try.

Let’s create the missing directory and then enable and start the service:

# mkdir /var/log/obhttpd
# sysrc obhttpd_enable="YES"
# service obhttpd start

As always I prefer to take a quick look if the daemon did bind the way I wanted it to:

# sockstat -4l | grep httpd
www      obhttpd    1933  7  tcp4   10.11.12.1:80         *:*
www      obhttpd    1932  7  tcp4   10.11.12.1:80         *:*
www      obhttpd    1931  7  tcp4   10.11.12.1:80         *:*

Looks good.

File Server option 2: Nginx

Next thing is installing Nginx and providing a working configuration:

# pkg install -y nginx
# vi /usr/local/etc/nginx/nginx.conf

Erase the example content and paste in the following:

user  www;
error_log  /var/log/nginx/error.log;
worker_processes  auto;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    keepalive_timeout  65;

    server {
        listen       80;
        location / {
            root   /usr/local/www/pxe;
        }
    }
}

This is by no means a perfect configuration but only an example. If you want to deploy Nginx in production, you’ll have to further tune it towards what you want to achieve. But now let’s enable and start the daemon:

# sysrc nginx_enable="YES"
# service nginx start

Time for the usual quick check:

# sockstat -4l | grep nginx
www      nginx      1733  6  tcp4   *:80                  *:*
www      nginx      1732  6  tcp4   *:80                  *:*
root     nginx      1731  6  tcp4   *:80                  *:*

Nginx is running and listening on port 80 as it should be.

File Server option 3: FTPd

FTP for you, eh? Here we go. FreeBSD comes with an ftp group but not such a user by default. Let’s create it:

# pw useradd -n ftp -u 14 -c "FTP user" -d /var/ftp -g ftp -s /usr/sbin/nologin

It’s a convention that public data offered via anonymous FTP is placed in a “pub” directory. We’re going to honor that tradition and create the directory now:

# mkdir -p /var/ftp/pub
# chown ftp:ftp /var/ftp/pub

If you intend to use logging, create an empty initial log file:

# touch /var/log/ftpd

Now we need to enable the FTP service for inetd (the “l” flag enables a transfer log, “r” is for operation in read-only mode, “A” allows for anonymous access and “S” is for enabling the download log):

# echo "ftp     stream  tcp     nowait  root    /usr/libexec/ftpd       ftpd -l -r -A -S" >> /etc/inetd.conf

As we intend to run a service that allows local users to log in via FTP, we need to consider the security implications of this. In my case I have created the “kraileth” user and it has the power to become root via doas. While OpenSSH is configured to only accept key-based logins, FTP is not. The user also has a password set – which means that an attacker who suspects that the user might exist, can try brute-forcing my password.

If you’re the type of person who is re-using passwords for multiple things, take this scenario into consideration. Sure, this is an internal server and all. But I recommend to get into a “security first” mindset and just block the user from FTP access, anyway. To do so, we just need to add it to the /etc/ftpusers file:

# echo "kraileth" >> /etc/ftpusers

Now let’s restart the inetd service (as it’s already running for TFTP) and check it:

# service inetd restart
# sockstat -4l | grep 21
root     inetd      1464  7  tcp4   *:21                  *:*

Ready and serving!

Optional: Checking File Server service

Time for a final test. If you’re using a webserver, do this:

# echo "TestTest" > /usr/local/www/pxe/test
# fetch http://10.11.12.1/test
test                                             9  B 8450  Bps    00s
# cat test
TestTest
# rm test /usr/local/www/pxe/test

If you’re using FTP instead:

# echo "TestTest" > /var/ftp/pub/test
# fetch ftp://10.11.12.1/pub/test
test                                                     9  B 9792  Bps    00s
# cat test
TestTest
# rm test /var/ftp/pub/test

Everything’s fine here, so we can move on.

What’s next?

In part 3, we’re finally going to add data and configuration to boot multiple operating systems via PXE.

Multi-OS PXE-booting from FreeBSD 12: Introduction (pt. 1)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/multi-os_pxe-booting_from_fbsd_pt1.gmi

This is an introductory article; if you’re familiar with PXE you will want to skip the excursion but may be interested in the “Why”. The article ends with the post-installation setup of my test machine, turning it into a simple router so that the next article can start with the actual PXE-related setup.

Also see part 2 here and part 3 here as well as part 4 here.

Situation

These days I decided to reactivate an old laptop of mine. It has processor of an older generation (Ivy Bridge) but it’s an i7 – and it has 24 GB of RAM, making it a somewhat nice machine for certain things. Problem with it is: The USB does not work (I think that the controller was fried)!

So how do I get a current OS installed on there as I cannot use the CD emulator I’d normally use? Sure, I could always burn a CD, but I don’t feel like it (unnecessary waste). I could open it up, remove the drive, place it in another machine, install the OS and then put it back. Nah, not too much fun, either. How about doing a network install then?

When I thought of that I realized that I had some old notes from about a year ago somewhere on my workstation. I originally meant to blog about that topic but never actually wrote the post. So here’s how to do what the post’s title says – updated to FreeBSD 12.2 and Ubuntu 20.04!

While I have a clear favorite OS and very much appreciate what FreeBSD has to offer for me as the administrator of a fleet of servers, there really is no reason to turn a blind eye to other Unix-like systems. For one thing it totally makes sense to always pick the best candidate for a job (which might not in all cases be one OS). That and the fact that you cannot judge which one is best suited if you don’t have at least some level of familiarity with various options. But in addition to that my employer runs a heterogeneous environment, anyway – and while I’m mostly occupied with the BSD side of things, there’s simply way, way too many Linux machines around to even think about avoiding them altogether all the time.

Why PXE-boot Linux from FreeBSD?

After an update, the old means of installing Linux servers as we do it at work had stopped working reliably. I looked at it briefly but soon found that too many things weren’t set up like you’d do it today. Therefore I decided that it might make more sense to start fresh. And while at it – wouldn’t it make sense to try and combine the Linux and FreeBSD PXE servers on one machine? That would mean one less server to run after all.

The next installation due was for a customer who requested an Ubuntu server. As a matter of fact we are transitioning to use that distribution more often for our internal infrastructure, too (decision by management, certainly not my choice!). For that reason one weekend I ended up doing something that I hadn’t done in a while: installing a fresh Ubuntu 16.04 system on a test machine I. After doing this I could write a whole post about how bad the installer actually is (static IP addressing, anyone??), but I don’t want this to turn into a rant.

So let’s just mention one single complaint: Installing takes quite a long time (I never understood why Debian and derivatives install stuff that they remove again later during the “cleaning up” phase…). Before the installation was even finished, I admittedly already had mixed feelings about this new system. But then this was what happened on the first boot:

[ TIME ] Timed out waiting for device dev-disk-by\x2duuid-0ef05387\x2d50d9\x2d4cac\x2db96\x2d9808331328af.device.
[DEPEND] Dependency failed for /dev/disk/by-uuid/0ef05387-50d9-4cac-b796-9808331328af.
[DEPEND] Dependency failed for Swap.
[  *** ] A start job is running for udev Coldplug all Devices (3min 34s / no limit)

You’ve got to be kidding! A freshly installed system going sideways like that? Sorry Ubuntu that’s not the kind of stuff that I like wasting my time with! I don’t even care if it’s systemd’s fault or something else. The boss “preferably” wanted an Ubuntu system – I gave it a try and it failed me. Ah, screw it, let’s deploy something I know of that it actually works (same hardware BTW, before anybody wants to blame that for a problem that’s definitely Ubuntu’s fault)!

I had a freshly installed FreeBSD with static IP configuration (which you probably want to use for a DHCP / PXE boot server after all) in a fraction of the time that the Ubuntu installation took. And it goes without saying: System starts as one would expect.

Excursion: PXE – An Overview

While there have been earlier methods for making a machine boot over the network, PXE (Preboot eXecution Environment) is what you want to rely on if you need to do that today. It is a specification widely implemented (in fact just about everywhere) and chances are very low that you will run into problems with it. Have a look around in your computer’s EFI or BIOS, often PXE-booting is disabled (and if you want to use it just once to install an OS on the machine, I recommend that you disable it again afterwards). How to make a machine boot over the net depends on its EFI / BIOS. Figure out how to do it four your metal and give it a try.

On your average home environment not too much should happen. The PXE environment will probe the network for the information that it needs, receive none and eventually give up. But what information does it need and which components are involved?

Well, it needs to load an operating system “from the net” – which means from another machine. To be able to talk to other hosts on the network, it needs to become part of said net. For this it needs to know the network parameters and requires a usable, unique IP address for itself. It can get all of that from a DHCP (Dynamic Host Configuration Protocol) server. If configured correctly, the daemon will accept DHCP requests and provide the client with a suggested IP as well as information about the network (like the netmask, default router, nameservers on the net, etc). It can also tell the client from where it can load the NBP (Network Bootstrap Program).

The latter is a piece of software, usually a special bootloader. It will be downloaded via TFTP (Trivial File Transfer Protocol). Think of it as very similar to FTP but much, much more primitive. The NBP is executed when the transfer is completed. It will then download its configuration file and then act accordingly, most likely fetching an operating system kernel and additional files, then booting the kernel.

TFTP is what the PXE bootcode speaks and uses. But due to its very simplistic nature, TFTP is not well fit for larger file transfers. Therefore such files are usually loaded via other means once the kernel has booted and more options are available. FreeBSD likes to use NFS, while on Linux HTTP is often used to receive a filesystem or installation medium image.

So the following services are involved when setting up a PXE boot server:

  • DHCP server
  • TFTP server
  • Webserver / NFS service stack

Preparing a FreeBSD router

Today I’m redoing my previous work in my home lab with FreeBSD 12.2. The machine that I’m using is pretty old (32-bit only Atom CPU!) but it has 6 Ethernet ports and still works well for network tinkering. I got it for free some years ago, so there’s really no reason to complain.

When it comes to the installation, in this case I’m going with MBR + UFS which I normally wouldn’t do for a amd64 machine. It’s a standard 12.2 installation otherwise with my “kraileth” user (member of the wheel group) added during the installation.

First thing to do is copying my public SSH key onto the server and then SSHing into the machine:

% ssh-copy-id -i ~/.ssh/id_ed25519 kraileth@hel.local
% ssh kraileth@hel.local

Now I switch to the root user, disallow SSH password-based login and restart the SSH daemon:

% su -l
# sed -i "" 's/#ChallengeResponseAuthentication yes/ChallengeResponseAuthentication no/' /etc/ssh/sshd_config
# service sshd restart

I don’t need sendmail on the machine, so off it goes:

# sysrc sendmail_enable="NONE"

Next is bootstrapping the package manager, installing the unbound DNS server and – for convenience – doas (a simpler sudo replacement). Writing a one-line config for doas is fine for my use case:

# env ASSUME_ALWAYS_YES=TRUE pkg bootstrap
# pkg install -y doas unbound
# echo "permit :wheel" > /usr/local/etc/doas.conf

Then unbound needs to be configured. To do so, edit the config file like this:

# vi /usr/local/etc/unbound/unbound.conf

Delete all the content, put the following in there (adjust to your network, of course) and save:

server:
        verbosity: 1
        interface: 10.11.12.1
        access-control: 10.11.12.0/24 allow

Keep in mind that this is not a production configuration! You will have to do some research on the program if you want to put it into production. Do at the very least read the more than thousand lines of comments that you just deleted (you can find it in /usr/local/etc/unbound/unbound.conf.sample). The above config will make the daemon bind on the IP configured via the “interface” statement and allow DNS recursion for any host in the subnet defined via “access-control”. Let’s enable the daemon now (so it will start after rebooting the machine):

# sysrc unbound_enable="YES"

During installation I chose the first NIC (em0) to be configured via DHCP as offered by my home router. This machine will act as a DHCP server for another network, so I assign a static address to the respective interface (em5) that I have connected to a spare switch I had around. It will also act as a gateway between the two networks it is part of:

# sysrc ifconfig_em5="inet 10.11.12.1 netmask 255.255.255.0"
# sysrc gateway_enable="YES"

To actually provide Internet connectivity to hosts in the 10.11.12.0/24 subnet, we have to enable NAT (Network Address Translation). Otherwise any host that is contacted via an IP in that range will have no idea how to answer, even though our gateway forwarded the message to it. NATing is done via the firewall. FreeBSD offers three of them. Pf is my absolute favorite, but ipfw is ok, too. I wouldn’t recommend ipf – it’s old and there is really no reason to use it if you’re not already doing so and don’t want to switch.

First we’ll create a configuration file:

# vi /etc/pf.conf

Now put the following in there (again: Adjust to your network) and save:

ext_if="em0"
int_if="em5"
localnet = $int_if:network

scrub in all
nat on $ext_if from $localnet to any -> ($ext_if)

This defines commonly used three macros: One for the external and internal NIC and one for the local network. It then turns scrubbing on (you almost always want to do this and I suggest that you look up what it does). The final line does the magic of the actual NAT.

You will want to define a more complex firewall configuration if you plan to use this gateway in production. But this is enough for my (demonstration) purposes. I suggest that you follow along and when you made sure that PXE booting works, tighten up security – then test if everything still works. Firewalling is a topic of its own, and in fact not a small one, though. If you’re new to it it makes perfect sense to do some reading. There’s lots of free material on the net as well as a great “Book of Pf” by Peter Hansteen if you prefer.

So let’s enable Pf on startup:

# sysrc pf_enable="YES"
# sysrc pf_rules="/etc/pf.conf"

Reminder: If you want to do things right, you will probably want to enable pflog, too, for example. This post is meant to point you in the right direction and to get stuff to work quickly. Familiarizing yourself with the various components used is your own homework.

It’s always a good idea to update the system to the latest patchlevel:

# freebsd-update fetch install
# freebsd-version -kr
12.2-RELEASE-p1
12.2-RELEASE

Ok, looks like kernel-related updates were installed, so it’s time to reboot:

# shutdown -r now

That’s it for the preparation work. Once the server comes back up again it should function as a router. If you configure a host with a static address in the 10.11.12.0/24 network, it should be able to have DNS queries answered as well as reach the Internet.

What’s next?

In part 2 we’re going to complete the setup of the PXE boot server so that a client can boot into Ubuntu Linux. The rest will be in part 3.

Exploring FreeBSD (3/3) – a tutorial from the Linux user’s perspective

This is the third and last post of a series of introducing FreeBSD to Linux users. You might want to take a look at the first post (talking about some things different from Linux) and the second one (about binary updating and package, user and service management) if you have not done so already.

If you’re all new to FreeBSD (or the BSDs in general) I tried to sum up the most important things to know about this OS family in another post. And if you want to know how to install FreeBSD (and what disklabels are as well as some other *BSD specific stuff), there’s yet another post dealing with it.

So what are we up to this time? There are a few topics left that I want to write about (and quite some more that I would like to touch on, too – but it doesn’t make sense to try and put too much into too little space): Updating binary packages, the ports system and updating “world” (the OS itself) from source.

Updating packages

In the last post we installed bash via FreeBSD’s port system (pkg). About one month has passed since then and a new version of bash has been released in the mean time (just as I hoped it would!). So let’s see how to update packages, right?

The most common case is that you want to update all your packages. There are two commands you should know in this regard:

# pkg update

This updates the repository catalogue so that the system knows which package versions are available in the remote repo. You don’t normally have to run this explicitly since FreeBSD will automatically fetch the latest catalogue if it thinks that the local one is too old.

# pkg upgrade

This will tell you which packages can be updated and perform the actual update if you choose to do it.

Updating binary packages

In this case, a new version of the package management tool was also released. Pkg must be updated before any other updates can happen but other than that it works just like any other update does.

The ports system

What are “ports”? The process of making a software (for which the source code is available) build on a system that it was not necessarily meant for is called porting. Depending on the piece of software this can be easy (the program builds out of the box) or extremely challenging (a lot of code needs to be patched to make it work). In order to make things easier for everybody, FreeBSD developed the ports system which is basically a directory for each application that was ported and a Makefile as well as some support files in it. These contain everything needed to build the respective application on FreeBSD simply by issuing make inside that dir. The directories make up what is known as the ports tree.

Fetching a port snapshot

The ports system originated in early FreeBSD and quickly spread to the other BSDs as well. And even on Linux there are people who like concept: Gentoo Linux for example is based on portage which builds on the very same concept (but works rather differently in the end). Well, since I told you to deselect the ports tree during the installation you do not have it on your system. So let’s first get it in place!

Getting the ports

All newer versions of FreeBSD offer the portsnap command which makes that very easy:

# portsnap fetch

If you do not have the ports tree on your system this downloads a snapshot, verifies it and also fetches any patches for ports changed after the snapshot was created. You can use the same command to fetch the newest patches if you already have a ports tree and receive any changes made in the meantime.

# portsnap extract

With this command you tell the system to actually unpack the snapshot and populate the ports tree. Only use this the first time you install the ports tree to your system. It doesn’t make sense to use it afterwards!

# portsnap update

You do not need this if you have just installed the ports tree for the first time. It is used to update the local ports tree after downloading any patches with fetch. If you wish you can also combine the two parameters to update the ports tree (portsnap fetch update).

Extracting the ports tree

You could also get the ports tree via Subversion. But portsnap is just so convenient to use that there’s barely any reason to do so.

Finding your way around the ports tree

So now let’s take a look at it! Where are all the files? They are in subdirectories of /usr/ports. We’ve installed bash in the last post using binary packages. Where would we find it in case we wanted to build it from ports? Being a shell, /usr/ports/shells/bash is quite a logical place, don’t you think? And where would you look for, say, the ruby interpreter? You’ll find multiple versions of it in /usr/ports/lang/ruby2x (ruby 2.0, 2.1, 2.2).

If you work with the ports tree for a while you’ll get at least an idea where things belong to. But what is the best way to locate a specific port? You can use the whereis command followed by the program name and it will tell you where the port lives! Just make sure you type in the right name. You won’t find php for example. But you will find the port if you look for php55 or php56 instead.

Finding applications in the ports tree

Still having trouble? Perhaps the page FreshPorts can help you. You can search there and chances are good that you find what you are looking for and can find out the category and port name that way.

Building from ports

The first question is of course: Why should you build programs from ports? The ports system was invented to automate the build process when there were no binary packages available and you had to build every program from source. Today you can easily work with FreeBSD without ever touching ports.

But when does it make sense to use ports? The simple answer: If you have special needs! The binary packages are pre-build and there’s no way to change any compile-time options. If you’ve ever manually build a program on e.g. Linux, there’s a good chance that you have met configure which takes options like –prefix=/usr –without-package-xz –enable-newest-feature and so on. If you need some program feature that the pre-packaged program does not come with on FreeBSD, you can use ports. Or if you do not want a certain feature built-in which is selected by default, you can also use ports.

Selecting build options for a port

For packages that can be built with different options which the author of the port thought were interesting, you will be given a nice dialog window in which you can select or deselect certain options. Just navigate into the directory of the ports tree where the files for the application you want to build live and issue make.

This will bring up the configuration window if there are any options to set. Please note that your selection will be saved so you are not asked the next time you build the port. If you changed your mind and want to reconfigure the options, you can use make config.

Building links from ports

If you order the port to “make”, the source code will be downloaded from a known location (you do not have to do this yourself), decompressed, probably applied some patches and then built. Once the build is complete, you can use make install to install the program and make clean to clear the build directory of files remaining from the built.

It is also possible to combine several commands which the program make takes (these are called targets and are defined in a file called Makefile). So you can build, install and clean up one port by issuing make install clean.

You also don’t have to worry about dependencies. If a port needs other programs (or libraries) which are not present on the system, they will automatically be built from ports, too. And one more important thing: Don’t hesitate to mix binary packages and ports on your system. You don’t have to choose one and stick with that all the time. In fact the ports produce custom binary packages which are then installed using the normal package system. That’s why pkg is aware of any program that you installed via ports and can for example remove it from the system if you tell it to. You could also go to the port’s directory and use make deinstall.

Recursive operations

If you want to build a complex program that has lots and lots of dependencies (like e.g. Libre Office), it is a good idea to let FreeBSD build it overnight. There is, however, a big problem that you’ll face if you try out large unattended builds: Every now and then, when a new port is built as a dependency, FreeBSD displays the configuration window and pauses until you make your choice…

This is why there are recursive targets: You can use make config-recursive and the ports system will go through all the dependencies and display the configuration. So you can select all the options that you need at once before you use just make to build all those programs.

Recursive source fetching

Mind one thing, though: If you enable more options, you may want to run config-recursive again. Why? Because the options that you selected may have pulled in new dependencies which are not yet configured. Running config-recursive will only display the configuration dialog for ports that were not configured previously. If you need to re-configure all ports, you can use the make rmconfig-recursive target to delete the stored configuration for the port and all dependencies and configure them again afterwards.

And in case you want to pre-load all source tarballs before starting an unattended build, there are the make fetch and make fetch-recursive targets. In very rare cases it can happen that all the sources that one port knows for its tarballs are no longer available (this is more likely to happen if you’re using a no longer supported version of FreeBSD and/or an out-of-date ports tree). You can fix this if you simply find another source of the needed file on the net and download it to the /usr/ports/distfiles/ directory where all those source tarballs for the ports live.

Updating the system from source

Just like with ports the first question ought to be: Why should you? And in this case the answer is even more: You probably shouldn’t. There are people who like to build from source and that’s ok. But if binary updates work for you, in general you should stick to them.

When do you need to compile the system from source? Well, obviously this is the method of choice if you are a developer who needs to build the absolutely newest code. But if that’s the case you’re probably not reading this tutorial anyways, right?

So – why should you do it? There are basically three main cases:

  • To have it done once
  • Because you want to aim for the stable branch
  • You want to customize e.g. your kernel configuration

Do not laugh at the first one. It is a perfectly valid reason. While building FreeBSD from source is extremely easy, it is good to have done it at least once. It will help you to get a little bit closer to your system.

FreeBSD comes in several branches. You can decide to follow another branch and compile the code for it. We’ll talk about that in a minute.

And last but not least if you have special requirements and want to customize your system for that. E.g. you man decide to compile your firewall of choice (FreeBSD offers three of them) into the kernel. In that case you have to build from source.

Getting the source

We cannot discuss scenario three (customizing FreeBSD) here. That would require its own post (or even more). Besides – I’m not too knowledgeable in that field.

Installing the certificate bundle

Let’s assume we want to follow the stable branch. First we need the appropriate source code. FreeBSD uses Subversion for version control and a slimmed version of it comes with the system (“svnlite”).

You may want to install the certificate bundle first so using a secure connection does not result in an error because the certificate is unknown. To do that you can simply use the following command: # pkg install ca_root_nss.

Next we need to checkout the current version of stable code with svn. FreeBSD source code always goes into /usr/src.

Checking out system source with svn

Start the checkout process with

svnlite checkout https://svn.freebsd.org/base/stable/10 /usr/src

and wait for Subversion to finish. This can take quite a while because the source code is quite large.

Once it’s done, you’re set. Go to /usr/src and issue make buildworld. This will build the userland part of FreeBSD (and – depending on your CPU – take a long time to finish).

System source checkout completed

What gets build goes into /usr/obj, btw. So the source code is kept separate from it and anything in /usr/obj can be easily removed anytime before doing a clean new build.

Building the FreeBSD userland from source

When the world build has completed, it’s time to build the kernel as well: make buildkernel – this does not take such a long time to complete.

Now both parts of the system need to be installed with make installkernel and make installworld. Always remember the correct order:

  1. Build world
  2. Build kernel
  3. Install kernel
  4. Install world

The reason is that “buildworld” needs to run first, is that it uses the system compiler to bootstrap the new compiler which is then used to build the whole userland and, after that, the kernel. And the reason that the kernel should be installed first is that after updating the userland you really should reboot. You’ll probably get away without rebooting if you just updated within the same release version but updating to a new release from source will mean that you cannot count on the system to just keep running like before due to incompatible changes made. In theory you are even encouraged to boot into single user mode to do the update! But I have not found that this is really required. Just mind the right order and stick to it.

Building the FreeBSD kernel from source

After rebooting you should find that the system is running on the new kernel. Now we’re on FreeBSD stable. However… That does not at all mean what you’re probably thinking it does!

New kernel is running

FreeBSD branches

I’ve stated before that there are multiple branches of FreeBSD, one of which is stable. Let’s take a look at what they are.

First there’s release. If you followed this tutorial along, version 10.1.0 was the system that we started with. Uname denotes the kernel as 10.1-RELEASE. Release is just that: A certain release. It will stay as-is forever, no changes applied to it.

Then there’s the patch branch or “releng“. This is “release + patches” and in fact the most stable branch available due to error corrections and security fixes. Uname will report something as kernel 10.1-RELEASE-p12. The patch branch is meant for conservative production systems.

We’ve already touched stable and even updated to it. If the patch branch is the most stable version, why is this one called “stable”? Yes, it is a bit confusing, I know. The reason is that this branch receives new features (which the patch branch does not) but the APIs are kept stable. Hence the name. This branch is not officially recommended for production use but the company that I work for has used servers with stable for years and they behaved absolutely fine.

Finally there’s current (called “head” in the repository). This is where the development takes place. If you’re not a developer or somebody who wants to test the newest features as early as possible, this is not for you.

What’s left

I would very much have liked to cover file flags & secure levels as well as jails. I’d liked to have written about tools like portmaster and system components like the three firewalls. But that might or might not happen in a future post…

What’s next?

In exactly one month I’m going to write my final exams to become a qualified IT specialist. So I’ll have to see what topic (if any) I manage to write about next month. Since I’ve always wanted to write the followup to my post about licenses, this may be a good candidate.

Exploring FreeBSD (2/3) – a tutorial from the Linux user’s perspective

This is the second post of the “Exploring FreeBSD” tutorial. If you didn’t do so already, may want to read the first part before this one. And if you are completely unfamiliar with FreeBSD, the posts installing FreeBSD and the general introduction of the OS may also be of interest for you.

In the first part we have configured SSH insecurely to allow root login, set up port forwarding in VirtualBox and briefly explored commands and the default shell on FreeBSD. Now it’s time to continue our journey!

Since some screenshots show a lot of lines I sometimes cut out the relevant part to save a bit of space.

Updating the system

FreeBSD is an operating system well-known for its reliability. But of course just like any other complex systems, it is not perfect – there are bugs and security holes. These are addressed rather quickly since FreeBSD takes those issues seriously. To always have the currently most secure and stable system you need to perform updates. As updating is important, let’s get right to it!

FreeBSD binary update

In FreeBSD there are several branches, but we will ignore this for now and save it for the last part of this series. There are also two supported methods of updating the operating system: Using binary updates and building from source. We’ll just cover the former here and also take a look at the later in the next post.

Updating your system within the current release basically boils down to issuing two commands (well, actually one command and two parameters): freebsd-update fetch and freebsd-update install.

Summary of files to update

Once the fetching is complete, freebsd-update will display a list of changes that will be applied to the system. There you can take a look at which files will be replaced with newer ones. If you agree with the changes, you can install the actual update and – if the kernel was affected as well – reboot.

Installing the update and rebooting

Updating the operating system is actually as easy as that. Keep in mind however, that with the BSDs operating system (“world”) and installed software (“packages”) are things separate from each other! We’ll cover updating packages in the next post.

Release upgrade from 10.1 to 10.2

But what if you want to update to a new release? We’re lucky here: Since this series of articles started with installing FreeBSD 10.1, a new release has happened: 10.2! So let’s update to the new release version, shall we?

Again the freebsd-update command is our friend. It can also do a binary upgrade from one release to another. It’s freebsd-update upgrade this time and with the -r 10.2 option we choose to upgrade to that particular release version.

The release upgrade needs a lot of patches and files!

This process takes a whole lot longer because there are of course far more files affected. But it is essentially the same process: Fetching patches, fetching new files and displaying three lists: files that will be removed, added and modified if the upgrade is installed.

After the upgrade: Rebooting again

After installing the upgrade, just reboot the system. A moment later you should be greeted by your new 10.2 FreeBSD system!

The updated 10.2 system

Adding users

User management is surprisingly easy on FreeBSD. There’s the powerful pw command that can do just about anything user related. And for adding users there’s also the convenient adduser script that makes adding users to the system a breeze.

Adding a new user

Most things are completely self-explanatory. What may however be new to you is the login class concept. Chances are that you won’t need them, but it’s nevertheless good to know that they exist and what they are. On a system with multiple users it could be that some of these users want to use e.g. different locale settings. FreeBSD allows you to create login classes that control localization – only affecting users who have that login class set for their account.

Take a look at the available shells. Missing something? In that case the shell is not installed on the system. If it was, adduser would offer it to you. All shells present on the system are recorded in /etc/shells by the way. Feel free to cat it out and compare it to what adduser offers!

Now let’s switch to the new user and try to become root again. Nope, sudo is not part of the base system. We could install it, but for now we’ll go without it. Fortunately we know the root password. So let’s su to root!

Only “wheel” members are allowed to “su”!

“Sorry”? Now what’s that? Certainly not a very helpful error message! Well, you just met another peculiarity of FreeBSD: You need to be part of the wheel group for su to allow you to become root.

So let’s try that out. And indeed: After logging out, adding the user to the group using pw usermod [username] -G wheel and logging in – su let’s us become root.

Package management

Traditionally programs have been built from source in an automated manner using something called the ports tree. We’ll cover that in the last part of this article series.

The other choice obviously is to use binary packages. FreeBSD has used what is used the pkg_*tools for a long time. Up to the still supported FreeBSD 9.x, these are the default package management utilities. You use pkg_add -r to add packages to the system, pkg_info to display information about them, pkg_delete to remove them, and so forth.

On FreeBSD 8.4 and 9.x the new pkg-ng tool could be optionally used. Since release 10.0 it is the new standard tool for dealing with packages. It is however not part of the base system and thus does not come installed by default.

It will however be binary-bootstrapped (a package manager is needed to install the first package, too, right?) if you first try to use it. For that process it doesn’t make a difference if you provide any parameters to pkg or not.

Bootstrapping the pkg binary package manager

Pkg-ng uses just the unified pkg binary which allows for subcommands. Once you have it on your system, you can use pkg install to add packages to your system, pkg info to view package-related information and so on.

Let’s just install the BASH shell for now. It is as easy as typing pkg install bash. Pkg will list the package and its dependencies and ask for confirmation. If you give it, it will download and install the packages. That’s really all there is to it.

Package management is of course an important topic. The new pkg-ng tool is so easy to use however, that I won’t make any additional examples here. Be sure to have a look at the man pages or read the handbook article.

Installing a familiar shell: Bash

User management again

Ok, we have BASH installed now. Just a moment ago I told you that it should now be available if you add a new user. Now what to do if our already created user should get it as the default shell?

What we could do is pw userdel on it and then re-create it. We could even manually mess with the /etc/passwd file. But this is not the best way to do it. In fact it is much simpler than that.

About to change the shell for the current user

There’s a program that let’s you conveniently change user information of existing users. It goes by multiple names and allows editing different user information. In our case we want to use chsh – “change shell”. It will fire up an editor and let’s you edit the user’s login shell among other things.

DO MIND that FreeBSD keeps packages separate from the base system! There is no /usr/bin/bash! If you enter that as the path to your login shell, the user won’t be able to login anymore. In FreeBSD you’ll find it in /usr/local/bin/bash. The same is of course true for other shells that are not part of the base system and for any other software that is installed from packages in general!

Altering user information

It is also noteworthy that you should NOT change the login shell for root. Leave it as it is and start bash manually if you really have to. On a toy system it may not be much of an issue, but there is no need to grow bad habits in the first place. If you ever need to repair a damaged system and you have changed the default shell for root, you have a good chance that it will bite you.

BASH is now the new shell for my user

System services

We have one more topic to deal with in this article. Knowing how to manage users and adding packages is nice, but being unable to mess with services kind of makes FreeBSD useless for you. So let’s take a brief look at how FreeBSD works with services.

You can simply ps -aux like on linux to take a look at what is running on your system. But how to manipulate daemons properly?

For a while now (it was introduced sometime in the 8.x release versions, if I remember correctly) FreeBSD comes with the service command. It is a valuable tool that you should start using right away. Sure, you can use the init scripts by hand, too. But service has the advantage that you don’t have to think if something belongs to the base system (and is thus located in /etc) or not (in which case you’d have to look in /usr/local/etc)!

Taking a look at services

It also provides a few more nice features. First let’s have a look at which system services are currently enabled (run automatically on startup). To do this, simply type service -e.

If you want to know all services on the system (which you could start), use service -l. This produces a long list, so it might be a good idea to use a pager like less or grep for something if you already know what you are looking for. In our example let’s look for the ntp daemon: service -l | grep ntp. No surprise: It’s called ntpd.

It won’t keep running without the right parameter because my clock difference is too big. But we’re not covering ntpd here, right? It’s just an example and you can of course use other services as well.

How to enable a service?

First let’s ask the system about ntpd’s status: service ntpd status. Now that error message tells us that ntpd is not enabled in the system configuration. It could still be started manually but as long as it is not enabled in the rc config file, FreeBSD keeps reminding you of that fact (which is actually a good thing).

Just like it suggested, we can use service ntpd onestatus. Truthfully it tells us that the daemon is not running. We can start it despite not being enabled in rc.conf using service ntpd onestart. Now onestatus lets us know that it’s running. Keep in mind however that such a manually started process will not be started when the system boots!

We could stop the daemon again using the service command. But to show off the init script way we’ll do it without it one time: /etc/rc.d/ntpd onestop.

And now finally let’s take a look at the configuration file and how we can enable any service. Fire up any editor on /etc/rc.conf and add the line ntpd_enable=”YES” to it. That’s all in fact. In case you want to give any parameters to the daemon, you can do so by adding an optional line like the following: ntpd_flags=”-x”.

Where services are configured: The “rc.conf” file

There’s a lot more to know about services and I encourage you to take a look at the FreeBSD handbook on that topic. But for our short introduction of the very basics, that’s it.

What’s next?

The next post will be the last one of the FreeBSD introduction. It will deal with the ports system, updating from source and a few other things.

Exploring FreeBSD (1/3) – a tutorial from the Linux user’s perspective

The previous blog post detailed the FreeBSD installation and the one before introduced the OS in general.

This one is going to show you around a little bit. There’s a lot to see here – so let’s jump right in!

Convenient access

Let’s start the VM with the fresh FreeBSD installation and login as root. The Virtualbox graphical console does allow you to work with your VMs but it is not a convenient way: You cannot copy just some lines of configuration into the buffer or paste a longer command to execute. Also if you’re like me and use an optimized keyboard layout, chances are that in VBox you’ll have to use US or any standard keymap.

Or maybe you want to scroll up your terminal when you need to see what that strange output five minutes again was. Sure, you can do that in a Virtualbox console, but it’s much more comfortable using the mouse. When you’d try SHIFT-PG UP or SHIFT-PG DOWN like on a Linux system it won’t work BTW. FreeBSD has its own scrolling mode which you can enable/disable pressing the scroll lock key.

And of course it’s always nice if you can see multiple logins at once – say with split-screen tmux. You can switch to another TTY with CTRL-ALT-Fx on FreeBSD just like in Linux. In Virtualbox you use right CTRL and one of the Fx keys. That works but you can’t see two or more logins at the same time.

Anyways: There are more than enough perfectly valid reasons to get rid of Virtualbox’s graphical console. So let’s just do that. But first we have to edit the configuration file for the ssh daemon:

# vi /etc/ssh/sshd_config

Find the line #PermitRootLogin no, remove the comment sign and change it to yes. Save and exit vi.

Now we’d have to restart the ssh daemon for the changed config to be applied. But we won’t do that. Instead we’ll just power down the machine. You can do that as you’re most likely used to from Linux:

# halt

A moment later you’ll see something like this:

All buffers synced.
Uptime: 50s

The operating system has halted.
Please press any key to reboot.

This is FreeBSD’s way of saying: “Alright, I’ve finished. You can safely power off the machine now”. Close the VM.

Note: To properly power off your FreeBSD machine you should use the shutdown command instead of halt or reboot as that will try to stop running services instead of just killing them. Shutdown can take the parameters -p to power down or -r to reboot the machine respectively. Thanks, TK, for reminding me of that as it really is something that a tutorial reader should know!

Now click on settings for your VM, choose network and click on advanced. Then click the button labeled Port Forwarding. In the new window add a new rule and enter 22 (the default port for SSH) as the guest port. Redirect it to any free port (higher than 1023) on your host machine – I’m using 20022 here.

Port forwarding rules for Virtualbox

Now let’s start the machine again. But since we don’t want the graphical console, let’s start the VM without it. Virtualbox offers something called headless mode which does just that: Start a VM in the background without a console for it. You could do so on the command line – but did you know that more recent versions also allow you to do this by double-clicking a VM while holding down SHIFT? Try it out!

Look at what Virtualbox calls the “preview” to watch the boot process and see that it is really starting. When it has booted up, it’s time to connect. User your favorite terminal to issue the following command:

$ ssh root@localhost -p 20022

You’ll then be prompted for the root password and then – there we are!

Alright! I don’t have to tell you that in general it is a very bad idea to enable root login for SSH – even more so if you allow logging in with password. But here for our purpose of just playing around on a non-production system it’s fair enough.

And what if you did something funny to your host’s ssh configuration and it won’t connect? Well, SSH problems are a special topic of their own. But here’s one not to miss: Whenever you run in trouble, be sure to include the “-v” switch for verbose output. This is bound to give you some hint at what is happening (and an idea what to search for with your preferred search machine if don’t know at all).

Welcome to FreeBSD!

That’s what your terminal is probably saying right now (if you’re just reading this post without following along, have a look at the screenshot).

Connected to the headless VM using SSH

Release notes and security advisories are probably not of too high importance for you if you’re new to FreeBSD. But I cannot recommend the handbook enough. It covers a large part of the system in detail and starts at a rather low-level (even for people without much command line experience). It is extremely helpful if you want to find your way around with FreeBSD. Have a look – and take your time reading. Really.

Don’t just jump at the forums. You are not ready for them, yet. Sure you’ll face problems. But this is Unix not Linux. kindergarten’s over. Learn to stand on your own feet and use your brain when you run into trouble. People in the forums will probably be happy to help you if you’ve read the handbook, FAQ, did some research on the net and still have no clue how to solve your problem. Do NOT consider it the first place to look for help! If the information you need is on the net and can be found easily then you’d steal everybody’s time asking stupid questions (and make yourself look like a dumbass at the same time).

Oh, and you may want to get used to consulting the man pages if you haven’t already. You may know that those exist from Linux but you may just have lived happily without them. They are a great resource of information. Embrace them and immediately benefit from the knowledge that they hold (without having to ask somebody and wait for an answer)!

Console commands

Use CTRL-L to clear the screen. Let’s see what FreeBSD can do! whoami, pwd, etc. You know the score. They all work and do what you’d expect.

Now let’s create two directories:

$ mkdir -p test/directories

Works like a charm. But perhaps we don’t really want them so let’s delete them again:

$ rm test
rm: test/: is a directory

Ah, stupid me. That command does not remove directories if it’s not given the -r switch! So let’s try again; the up-arrow key brings back the previous command so I can just append the missing switch. Now that should work, shouldn’t it?

$ rm test -r
rm: test/: is a directory
rm: -r: No such file or directory

What the…?! Why is stupid rm interpreting the switch as a file? This should work, right? Nope, rm is reacting just fine, it’s really my bad. But it does work on Linux, doesn’t it? That’s correct. So what’s happening here? In fact the above line is syntactically wrong. You should always state: 1. command, 2. option(s), 3. parameters. When rm met the first parameter (“test”) here, it stopped expecting options and thus treats “-r” as another parameter. And that’s fine – there could be a file “-r” after all as that’s a perfectly legitimate file name!

$ rm -r test

No surprise here: This works.

Lesson 1: FreeBSD is much more strict while Linux (with its GNU tools) lets you get away with a lot of strange things. Some people say that this makes GNU/Linux more “friendly” but that totally depends on how you look at it. What if you want to remove the files “test” and “-r” in GNU-land? It’s possible there, too. But you have to type this:

rm -r -- test -r

The double dash tells rm to stop parsing options and treat every following input as a parameter. Works just as well, for sure. But you need to know how to use the double dash or you can run into pretty bad edge cases without a clue what’s going on. The BSD way of doing things is not aiming at this kind of “friendliness” but just doing things right in the first place.

Mind the syntax!

Let’s take a look at what df can tell us. Except for the device name there’s nothing too strange here. If you wonder how that device naming works, you may take a look at my previous post, where I explained it in section “Excursion: Partitions and *BSD”.

But there is one special thing about df which you might run into if you decide to do some *BSD: Capacity can go over 100%! Yes, you can come across e.g. the root partition being at 105% or something like that. This is not an error in df – it is something that is conceptionally different. FreeBSD reserves 8% of disk space for the OS and root and df considers the complete space minus the reserved space to be 100%.

The shell

Now let’s try to play a little with environment variables. Setting and displaying one should be easy, no?

$ export ALL_FINE=”sure!”
export: Command not found.

Uh, what’s happening here? The system doesn’t know about export! Do you know what it belongs to? Try running which export on any Linux system. You won’t find it there either. What a mysterious case! Ok, not really. Export is an internal command provided by bash, the standard shell of most Linux distributions. FreeBSD uses another default shell, so let me introduce: The C-Shell (csh)!

Different shell: csh.

The most important thing that you need to know about the C-Shell is that it is fundamentally different in many ways. I’ll just show setting an environment variable here to show off something; the whole topic is huge and if you care enough, you should easily find something helpful to read. But we’ll simply be installing bash later in this tutorial. Why? Because there’s enough to learn for FreeBSD anyway and it doesn’t hurt if people can at least explore with a shell they know and feel comfortable with.

$ setenv ALL_FINE “sure!”

This is what the csh expects if you want to set an environment variable. Mind the missing equals sign! And if you want to display all the variables currently present, just run setenv without any parameters.

What’s next?

This post tried to give some very basic information for you to get along on FreeBSD systems. In the next two parts we’ll have a look at user, package and service management, system configuration, ports and updating the system.

Installing FreeBSD – a tutorial from the Linux user’s perspective

This article deals with installing FreeBSD. It’s meant for the first time FreeBSD user who has a bit of Linux background.

The installation process is actually quite simple and straight-forward. I’ll guide you through in some
detail, anyways, because I think that there are a few things that are good to know even if they are not strictly necessary.

On a side note: My blog turned three a few days ago. I decided not to write a birthday post this year. Let’s just get on with some real content!

Preparations

The easiest way to try out a new operating system these days is to run it in a virtual machine. I’m using VirtualBox to create the screenshots and you are invited to create one, too, and follow along. Just reading the post is also fine, though. And of course nobody will stop you if you decide to dedicate a real machine to this project because you have some spare hardware around.

If you opted for VirtualBox, you just need to create a new VM first. Select FreeBSD as the OS and give it a bit more RAM and far more disk space (since we’re actually going to put our new system to some use) than VirtualBox suggests. Other than that, you’re set up. You should not need to make any other changes (though you can of course adjust things if you feel like it).

Now download an ISO image if the installer CD from here.

We’ll be using FreeBSD 10.1-RELEASE. Pick the right one for your CPU architecture – if you’ve got a PC then i386 means “32-bit” and amd64 means “64-bit”. It doesn’t matter if you’ve got an Intel or an AMD. Make sure you choose an installer image and not a virtual machine image since we want to install FreeBSD after all!

The smallest image is fine; that’s FreeBSD-10.1-RELEASE-amd64-bootonly.iso for a 64-bit system (or the .xz one which you need to decompress after the download). If you were going to install FreeBSD on several machines or on one that is not connected to the net, it would make sense to choose a bigger one which actually includes the OS you want to install. But for our purpose the bootonly (which will download the OS files) suffices.

Got everything? Excellent. Start up your VM and put the installer image into the virtual drive. Then you’re set to go.

Installing FreeBSD: Keyboard and file sets

You should be greeted by the bootloader. Either wait ten seconds or just press the Enter key (we don’t want to set any special options at this early stage).

The FreeBSD bootloader screen

Now the kernel will probe your hardware and the system will come up. Once everything is done, the installer will be started automatically. It’s not the most beautiful installation program ever, but it does the job. And it actually makes the installation really easy. Until FreeBSD 8 there was the old installer, BTW. It was more powerful but the installation was also more complex (and it was even more ugly! ;)).

Meet the FreeBSD installer!

After choosing “install” you can select a keymap or go with the default one (US). It’s all up to you here.

Keymap selection in the installer

Then we have to give the virtual computer a name. I decided to call the machine beastie which is the name of FreeBSD’s mascot (the little red daemon with the fork).

Setting the hostname of the machine

Next select which optional sets to install. You won’t need doc. This may have been very useful in the past but today you’ll probably want to simply browse the documentation online instead of locally, anyways. If you really want it, it won’t hurt you, though, but you could just as well save yourself (and the FreeBSD project) some bandwidth.

Games is not what you probably think, BTW. It’s pretty obvious that this won’t install DooM or Quake or the like. But it’s not even something like Solitaire or Freecell that comes with Windows. It’s in fact just a bunch of command-line based games, the most popular being “fortune” mentioned by the installer. It is a “fortune cookie simulator”. Most people probably won’t call any of the programs from this set an actual game. They are more or less just little fun or joke programs. Actually they have a long tradition with Unix and they are very small, too. For that reason they are included in a FreeBSD install by default. You may deselect this set if you really want but they definitely won’t hurt you, either.

Choosing the sets to install

The lib32 set contains the 32-bit libraries which are needed if you want to run 32-bit executables on a 64-bit FreeBSD system. There are actually more cases where you need them than you might think. So if you aren’t really, really short on disk space or know exactly that you won’t ever need them (hint: When you are just starting with FreeBSD, you don’t) just leave this one checked.

For our tutorial please make sure you uncheck the ports and src sets which contain the ports tree and the system source code respectively. We will need both later – but we’ll fetch the current version of both them using different a different means!

Installing FreeBSD: Network

Now you need to configure your network so that the sets can be downloaded. This is a pretty straight-forward thing to do: Select your NIC (chances are you only have one, anyways), select IPv4 and choose DHCP if you wish to automatically receive an IP address (otherwise configure the NIC manually). You probably don’t need IPv6 so skip that unless you’re really actually using it in your LAN. If you selected DHCP, FreeBSD should have received the address of at least one DNS server as well. If it didn’t, make sure to provide one. Otherwise name resolution won’t work and the installer cannot download the file sets.

Selecting the mirror

The next step is to select a mirror server from which to download. It obviously makes sense to choose one that is located in a place near to you as it is more likely to provide a good connection for you.

Installing FreeBSD: Disk layout

All that’s left is setting up the hard disk(s). The basic choice you have to make is which file system you want to use. FreeBSD basically supports two native file systems: The traditional UFS (“Unix File System” aka. FFS or “Fast File System”) and the next-gen filesystem ZFS. The later is a sophisticated FS with a lot of interesting features. For quite some people, fact that ZFS is considered stable on FreeBSD is a killer feature of this operating system and even the main reason why they choose this OS.

ZFS is however well beyond the scope of this post. A dozen of posts like this could be written on that topic (probably not by me, though, since my ZFS knowledge is rather basic, at least at the time being).

So we’ll opt for UFS now. Partitioning a FreeBSD system is a little bit more complex than partitioning Linux. Fortunately there’s the Auto (UFS) option in the installer. We select that and want to use the entire disk.

Disk partitioning

Today there are two partitioning schemes in use on the PC. FreeBSD defaults to the newer one, GPT. You could also use the older MBR instead if you have any reason to do so. And in fact you could even go without any of them! But that’s going deeper than we need right now.

The installer suggests a default partitioning: A boot partition, one for the root file system and one more for the swap space. Depending on what you want to do with your FreeBSD machine, this is most likely not what you want. But for our test system it’s fair enough.

The default partitioning layout

After hitting “Finish” and “Commit” the changes are written to disk and the actual installation begins.

Excursion: Partitions and *BSD

Partitions are a topic which can be highly confusing for the beginner (at least it has been for me before I did some research in this area). The problem here is that in the FreeBSD world a partition is something different what you might think. And to make matters worse, the terminology differs even between the BSD distributions!

Most Linux distros use the old MBR (“Master Boot Record”) partitioning aka “DOS partitioning”. Same thing if you come from a Windows background. You know the score: Up to four primary partitions and extended partitions if you need more than that. Chances are that your Linux distribution uses three primary partitions: One for /boot, one for SWAP and one for /. If they are on the first hard drive (sda) of your pc then they will be called /dev/sda1, /dev/sda2 and /dev/sda3.

These partitions are known as disk slices in the FreeBSD terminology. So it’s just a different name for the same thing, right? That cannot be so bad! Wrong. The MBR partitions are the same thing as the slices, yes. But the fun starts when you learn that the BSD systems use a mechanism called BSD disklabels. These divide the MBR partitions further and if you think in the DOS terminology of partitions, disklabels actually allow for what might be called sub-partitions as that is what they are. Unfortunately these are just called partitions in the FreeBSD context!

So remember this: In FreeBSD a “partition” is what you may think of as a sub-partition and a “disk slice” is what you commonly know as a partition. How come that we have all this confusion? Who’s guilty of causing it? Well, things are not so easy here…

Unix began its life not on the PC but on bigger research computers which did not support partitions at all. So the Unix people came up with disklabels to partition disks into up to 8 partitions. Since that is what they are, it was an obvious choice to call them partitions. Quite some time later the PC platform supported MBR partitions which were also called thus. The real trouble started however when Unix was ported to the PC: Now there were two different things with the same name! For compatibility’s sake, FreeBSD embeds its partitions (sub-partitions, remember) into what they call disk slices (MBR partitions) to be able to distinguish between the two.

As a consequence, FreeBSD needs only one disk slice (MBR partition) because it can create partitions (sub-partitions) on it, e.g. for SWAP.

If you are using the MBR scheme, it leads to a naming like e.g. /dev/da0s1b. This means the first SCSI disk, slice 1 (primary MBR partition 1), partition 2. /dev/ada1s5f means the second SATA disk, slice 5 (extended MBR partition 1), partition 6.

FreeBSD can also do without slices. This is called “dangerously dedicated” mode. The name sounds quite worrying but in fact the only “danger” is that it is highly unlikely non-BSD systems will be able to read any data from it (since they don’t know about disklabels). If you ever come across something like /dev/da1d, you know that it’s the 4th partition (sub-partition) on the “dangerously dedicated” (MBR partition-less) disk da1. If you just have *BSD on your drive, you can use this mode and there’s no “danger” for your data.

Fortunately things became easier with the introduction of GPT (“GUID Partition Table”), a newer partitioning scheme that supports more than the 4 primary partitions of MBR. And since more than enough partitions can be created this way, FreeBSD does not use disklabels to subpartition them further. The good thing is that many other operating systems know GPT partitions, too. The bad thing is that we have another kind of partition that’s just called… Partition.

Newer FreeBSD installations default to GPT partitioning. If you look for your drives then, you’ll probably find them as something like /dev/ada0p1 which means the first GPT partition on the first disk.

If you have to use the older MBR partitioning for some reason, there are a few things you should know about disklabels. Disklabel a is meant to hold the root filesystem (/), b is for SWAP. C is completely special; you cannot use it as a normal partition. It’s always there and covers the whole disk. This is used by some tools to access the disk in raw mode, neglecting any partitions or whatever. Historically d stood for the whole disk and c for the complete slice – but that time has passed.

That’s a lot of information, I know. But you’ll get the hang of it if you want to. It’s not that difficult once you got rid of the confusion.

Installing FreeBSD: Putting the system on the disk

Now lean back for a while; the installer will fetch the distribution packages from the net first.

Fetching the distribution packages

As said before, this step is skipped if you use a bigger image. In that case you’ll also configure the network settings later in the installation process.

Installing FreeBSD: Final steps

When all distribution packages are downloaded and extracted, you are prompted for a password for the root user.

Setting a password for the root user

Then you have to tell FreeBSD about your time settings. If it is the only OS on your machine or it shares it with other Unix-like systems like Linux, go for UTC. If you also have Windows on the same computer, however, be sure to select No here. Windows and UTC is a mess.

Choosing the time setting

In the next screen you can choose which daemons should be started during the system boot process. Unselect dumpdev, it’s of no use to us (if you were able to read crash dumps and debug the applications with this info, you wouldn’t really read this post now, would you?). Keep sshd selected if you want it.

Selecting the daemons for autostart

Choose not to add any users right now. We’ll be doing this later and learn a bit about user management on FreeBSD! Next hit “Exit”, tell the installer that you don’t want to make any final changes.

Exit the installer

Choose to reboot. And that’s it.

Reboot to finish the installation

If you remove the installer image, your machine should boot into your new FreeBSD system. Welcome on board, new BSD user!

What’s next?

The next blog post will deal with some of the very basics of a fresh FreeBSD system.