Exploring the CBSD virtual environment management framework – part 1: Introduction and installation

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2022/cbsd_pt1.gmi

One interesting feature introduced in early Unix (V7) is the chroot(2) syscall (accompanied by the chroot(8) command added later). They allow for putting processes in what is called a chroot jail. What this means is that for the chrooted process (and its children) the filesystem root is not the one of the actual host system but some directory further down the filesystem hierarchy. The result is restricting access to that directory and all the dirs below it and hiding away everything above. This allows for some simple sandboxing and is often used e.g. with FTP where users are chrooted to their home directory.

It is noteworthy that the SysV line of Unix (plus SysV-inspired Linux) and the BSD line follow a different philosophy regarding chroot jails. According to the former it is not a security device and breaking out of a chroot is a documented feature. BSD in contrast was determined to make chroot jails useful for increasing security and tried to prevent escaping from them.

Ever since Poul-Henning Kamp implemented FreeBSD jails in 1998 as a more solid and more flexible means, this legendary feature allows for what has later been characterized as OS-level virtualization or containerization. Unlike simple chroot jails they are not limited to filesystem restrictions but also do process isolation, have their own IP address and so on. Since they were introduced, many new features have been added. Today jails support multiple modes of isolation of SysV IPC, can be delegated ZFS datasets and may even have their own network stack (thanks to VNET). It’s pretty obvious that such a powerful tool is not exactly trivial to master. And while you can create and operate your jails with just the FreeBSD base system, people have written a whole bunch of utilities to assist in using jails successfully. Those applications are called jail managers. This article covers CBSD as of version 13.1.19.

Disclosure: I have been in contact with the CBSD maintainer for quite a while now and we also both participate in the Advance!BSD project. So if you were hoping for a completely unbiased look, I’m sorry (I’m trying my best, though). On the plus side this means the information you can find in here can be considered reliable.

Part 0 of this series is a general discussion of what virtualization actually is and what you should know to make best use of what will be covered in the later parts.

What is CBSD?

There are multiple jail managers to choose from (and there have been many more which have been abandoned after a while), however CBSD is rather unique. After originally being used internally at a small game development company, it was open-sourced and first released to the general public in 2013 by Oleg Ginzburg. This makes it the oldest jail manager project that is still active (and which has been in continuous development for almost 10 years now!). This certainly is quite an achievement, yet it’s not even one of its main selling points. While jail managers usually are conceived to allow for more convenient jail operation on a single node, CBSD was designed with clusters in mind! In fact, the “C” in the name refers to clustering. Which of course doesn’t mean that it cannot be very useful on single nodes as well (which is what we will of course focus on in the first parts of this series).

CBSD is also much more versatile compared to common jail managers. You have surely noticed that it is being called a “virtual environment management framework” in the title. No, this is not just a fancy name for a jail manager! CBSD is simply more than that. Sure, it does jails. However it manages Bhyve Virtual Machines, too. It has support for Xen as well. It can manage Qemu VMs for you and will even lend a hand with VirtualBox if you insist. It does all of that in a mostly consistent way (there are places where it shows that it originally started with just jails). Its flexibility does not end with all of those domains that it works with. CBSD also supports multiple ways of management: Command-line driven, via dialog(1) text UI menus, Vagrant-style (CBSDfile), using an API and more!

At this point you are probably not surprised to hear the it does not stop at creating, destroying, starting and halting your virtual environments but assists with clusters (failover setups, migration and so on), too. CBSD provides tons of options to setup your environments as you need them. It supports resource limitation and accounting. It provides powerful means for automated deployment within jails as well as many other things.

The project is native to FreeBSD and is able to use optional features if their requirements are met (e.g. it’s running on ZFS-backed storage). It is not limited to that OS, though. CBSD feels at home on the HardenedBSD variant as well and for example supports the special update process of that platform. It will happily manage your jails and VMs on DragonFly BSD (and supports HAMMERFS features), too. Support for VMs on NetBSD and OpenBSD might be added in the future if there’s interest shown in it.

So to put it short: It’s your Swiss Army Knife when it comes to any virtualization needs on the BSDs. Fasten your seat-belts, we have a lot to explore. And it will be a wild ride for sure.

Installation

CBSD is in ports on DragonFly BSD, FreeBSD and HardenedBSD. But unless you require special firmware to support GPU passthrough (which is currently the only port option), you may want to just install from packages:

# pkg install cbsd

CBSD can also be installed as a package from Ravenports on both DragonFly BSD and FreeBSD. Right now this is discouraged as the port was only added recently and there are still various problems (due to the non-standard prefix) that need to be ironed out. If you’re interested, check the date – it’s likely that the port will be in good shape before too long (probably in early 2023).

# ravensw install cbsd-single-standard

The ports have few dependencies and are very quickly updated when there is a new release. If you plan on contributing to the development, you may later want to clone the repository from GitHub and use that. The readme file includes the steps needed to do so.

After installing the package, CBSD needs to be set up. The installation message tells you how:

cbsd is now installed, but requires additional setup steps:

Upon initial installation, run:

env workdir="/path" /usr/local/cbsd/sudoexec/initenv

Where /path is the path to the cbsd workdir, e.g.: /usr/jails

For upgrading an existing installation, run:

cbsd initenv

We’re going to do this next time. For now let’s first understand how CBSD is structured on the filesystem.

CBSD filesystem organization

As you might have guessed from its many features, CBSD isn’t exactly a trivial tool. There’s *two* places that you need to know about.

If you install CBSD, its files will spread across several places in your filesystem. The package will populate (note: if you installed CBSD from Ravenports, a custom prefix (i.e. /raven) is used instead of /usr/local by default!):

• /usr/local/bin (the main cbsd executable)
• /usr/local/cbsd (this is where the main load of its scripts, profiles, modules and other data goes)
• /usr/local/etc/rc.d (CBSD comes with 5 init scripts at the time of me writing this article)
• /usr/local/libexec/bsdconfig (you guessed it: CBSD plugs into bsdconfig!)
• /usr/local/man (main cbsd(8) manpage – the others are in CBSD’s main directory)
• /usr/local/share/licenses (just what the name says: CBSD’s license (2-Clause BSD) is in here)

This directory scheme was chosen by the developer and the maintainer of the port according to FreeBSD’s standards. If for some reason you really cannot live with that this means that you will have to package the program yourself.

The other one is CBSD’s working directory and this is where you get to choose the location (but of course not the structure).

CBSD working directory

So what is the workdir and what will it contain? The example in the install message is /usr/jails which is reasonable if you plan to use jails only. I frequently use both jails and Bhyve VMs, so I usually create a dataset directly below the filesystem root: /cbsd. Here’s what it looks like:

% tree -adL 1 /cbsd
/cbsd
├── .rssh
├── .ssh
├── basejail
├── etc
├── export
├── formfile
├── ftmp
├── import
├── jails
├── jails-data
├── jails-fstab
├── jails-rcconf
├── jails-system
├── modules
├── nodes
├── share
├── src
├── tmp
├── var
└── vm

20 directories

That’s quite a bit of stuff there. Let’s discuss some of the important ones before we’re even setting up CBSD.

• In the basejail directory are – surprise! – the basejails. This is what it looks like on my workstation:

% ls /cbsd/basejail 
base_amd64_amd64_12.3 base_amd64_amd64_13.0 base_amd64_amd64_13.1

On this machine I can quickly create jails with a 12.3, 13.0 or 13.1 userland. If I wanted to create a 12.4 jail (as that’s due to be released in a couple of days as of when I’m writing this), a basejail for it would need to be created first. A basejail is literally the extracted content of the base.txz distribution set of the respective version in most cases.

• .rssh is the directory where private keys for controlling additional nodes go when working in cluster mode.
• In .ssh, you will find private and public SSH keys for the local node as generated during initenv. The key file name is the md5 sum of the nodename.
• The etc directory holds configuration and profile files. But hey, not so fast! Weren’t those supposed to be in the /usr/local/cbsd/etc?? Yes, and in fact that’s where most of them are copied from. And before you’re running away screaming, here’s why: CBSD supports sophisticated usecases where you may want to have multiple working directories, each with their own specific configuration values. You don’t have to use this, but in case you need that extra flexibility, it’s right there.
• The export directory is empty by default. When you create images of your jails they will end up here.
• The formfile directory is where files related to the forms functionality go. We will explore them later in this series, but if you’re curious: It’s how CBSD allows for external configuration management solutions (like Puppet) to aid in the automated deployment of software in containers.
• When you use the text UI dialog menus to create e.g. a jail, the choices that you made are recorded in a configuration file. Those get stored in the ftmp directory for later use.
• import is where you put your images created via export before importing them again.
• The jails directory contains the mount points for jails with a read-only base system.
• jails-data: Below this is where the filesystem root for each of your jails live. Want to access any of your jails files from the host (or e.g. copy or rsync some data over from one jail to another)? This is where you need to look. If for example I want to edit the /etc/rc.conf in jail1 from my host machine, the file to go for will be /cbsd/jails-data/jail1-data/etc/rc.conf. Mind the beginning of that path as you probably chose a different workdir.
• You can use jails-fstab to mount additional paths from the host into the jail.
• jails-rcconf is a legacy configuration directory from the time before CBSD used SQLite databases. You could use it to store rc.conf files in there that affect jail startup. If you’re not a long-time CBSD user who has very old jails that used this, don’t start using it now. It’s completely obsolete and only still there for compatibility reasons.
• The jails-system directory is a place where CBSD stores special information for each virtual environment. Those include e.g. resource limits, traffic stats and additional helper scripts.
• modules – who would have guessed it? – is where modules go. Modules are helper scripts that allow CBSD to perform actions of various kinds like package, service or user management.
• The nodes directory is used by CBSD when running in cluster mode to store information. You should probably leave this alone.
• The share directory could also have been called “misc”. It holds information about OS git branches, skel directories and other data that doesn’t fit well anywhere else.
• In the src directory you will find all the “source” files that CBSD downloaded. There will be the ISOs that you use to install various operating systems or Linux distributions in VMs. There will be the etcupdate tarballs for jails you created from base.txz distribution sets and the entire source code for those that you populated via the buildworld method.
• It comes to no surprise that tmp holds temporary files: DHCP leases, lock files and such. You may also want to leave this one alone. CBSD knows what it’s doing with it. Do you?
• The var directory contains a hierarchy structure much like any *nix system’s usual /var. Most importantly here is var/db: There’s several SQLite databases in here. Especially the one symlinked local.sqlite is a great choice to be backed up regularly if you care for your managed resources – and in particular before you try to “do something” which CBSD does not have an option for!
• In vm finally CBSD keeps data exclusive to Virtual Machines: There’s e.g. the bhyve config for every single VM, you can find a log file that holds VM start output (useful to diagnose problems), more databases and the VM’s virtual disk (which may be a symlink to a ZVOL).

As you can see, there’s plenty going on here. Also you should have gotten an idea that the workdir is the kind of directory / dataset that can take quite a bit of space (depending on what you plan to do). So especially if you’re using UFS: Choose the location of your workdir carefully! That already mostly filled-up partition over there might not be the ideal candidate – or it might, if all you want to run is a few small jails. Make use of ZFS if you can (ok, that’s more of a general rule with FreeBSD and most probably one that you already follow!). Do some more planning ahead otherwise.

Your workdir is not carved in stone, though. As hinted above, you can even have multiple. However working with multiple workdirs not because you want them separated but due to space constrains is one perfectly avoidable source of frustration.

What’s next?

You should now have a pretty good idea of what CBSD can do for you and know a bit about it’s structure. But we haven’t done anything with it, yet. Actually, at this point we couldn’t even do so! Next article will describe the setup process and the choices you need to make there before we finally start creating jails.