Exploring the CBSD virtual environment management framework – part 3: Jails (I)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2023/cbsd_pt3.gmi

This article is about using the dialog menus to create a jail, start it and enable SSH in there so the jail does something.

Part 0 of this series is a general discussion of what virtualization actually is and what you should know to make best use of what will be covered in the later parts.
Part 1 covered an introduction of CBSD as well as the installation.
Part 2 detailed CBSD’s initial setup process.

Composing a new jail, dialog-style

Alright, for part 3 we are finally able to put CBSD to good use. Again, a FreeBSD 13.1 test system is used, CBSD is version 1.13.21 and /cbsd was chosen as the workdir.

After two entire articles without managing anything with a management framework… *drum roll* …we’re going to create our first jail using CBSD. Yes, finally! There’s at least two more topics that I’d like to discuss before getting into jail management, but I also don’t want to delay things further. Just so that you’ve heard it already, CBSD ties into bsdconfig(8) as a submodule. We’re going to explore this a little later. It’s good to know, though, in case you are in need for an easy way to change the configuration you made during the initialization.

In CBSD there are often multiple ways to do something. There certainly are several when it comes to creating jails. We’ll start with what is the easiest method and the one that I would recommend to newcomers: Using a dialog-style menu. CBSD provides a nice, curses-based Text UI, building a menu for you to navigate. Since this article series assumes that you’re just getting started with the tool, it is the perfect choice for now. It allows you to get an idea of what the options are that you can or should tweak for your jails. Later you may find that other means of achieving the same goal are more useful as they are way quicker to operate than navigating a menu.

CBSD jconstruct TUI version

But let’s get to it already! Run this command:

# cbsd jconstruct-tui

Nice menu, eh? You can absolutely give this to people who are not FreeBSD admins (I’ve done that and it worked perfectly well!). Most of the options here are self-explanatory, but we’ll just go over everything quickly (at least in this main layer).

Jail profiles

Profiles are in fact exactly what you probably thought: Presets of options for various types of jails. You can completely customize any of them, but presets allow you to quickly select what is closest to what you want to do. For the beginning you may want to stick with the plain and boring default jails. The vnet profile is for you if you want to create jails that have their own network stack and e.g. allow for running their own firewall instance. If you want to try out something that you know needs options set that (for a good reason!) are off by default, give trusted a spin. For example if you need to allow for raw sockets because you want to use ping(8) in a jail. I hope you know what you are doing, though. And in case you really need a Linux jail (you have my sympathy!), those profiles may be a useful shortcut.

Selecting packages

Choose pkglist if you want CBSD to install packages inside your new jail for you. If you do, it will start indexing and preparing the available packages of every category for displaying via a menu.

Selecting packages from the ‘archiver’ category

I do not really use this feature because it’s usually easier to just use pkg(8) inside the jail (or pkg -j on the host). I would definitely consider the menu if I were creating a template for various jails, though.

Setting the jail name

Next choose a name for your jail. This is a reference for internal use only, so unless you like typing long names, select something short (but ideally meaningful enough to you to know what that jail does). CBSD suggests ‘jail1’ which is fair enough as an example but a pretty bad name otherwise. Find a scheme that works for you and use that.

Whatever you go with in the end, do mind what CBSD tells you here! Stick with letters and numbers. Do not try anything fancy. You may find that you seem to get away with disregarding CBSD’s warning. And indeed: You can do that. Things may even actually work for a while and you start laughing at stupid, conservative CBSD. But then things break. I’ve had to deal with the extremely weird effects of this after a customer forgot about the restrictions one time and put another character in there. It wasn’t pretty. Nothing valuable was damaged, but we lucked out there and coming clean again was some work, too. So do your future self a favor: Stick with letters and numbers, period. And if you insist on living dangerously: At the very least avoid any character that may cause trouble in regard to shell variables.

Setting the jail host name

Now it’s time to choose the host name for the jail. CBSD will propose something based on the jail name, but with an invalid TLD. So you’re clearly meant to fill in something sensible. Free to enter whatever you want to use as the hostname (as long as you own the domain, of course!).

Defining the IP address to use

You need to set up your jail for networking. While the variable is called ‘ip4_addr’, this is for historic reasons only. Entering a v6 IP address here is perfectly fine and supported.

In rare cases when the jail does not need any network connection whatsoever, set this to ‘0’. Mind the difference between ‘DHCP’ and ‘REALDHCP’ here! The latter is what you probably suspected the former to be. What ‘DHCP’ means in this context is this: Let CBSD pick a free IP address from the pool that you assigned it to use (during the initial setup, you remember?).

Set this as your use case requires. E.g. “DHCP,DHCPv6” would assign an IP from the IPv4 pool of addresses as well as from nodeip6pool if it was configured. Manually defining it like “,2a05:3580:d811:800::2/64” works, too. We’ll come back to this when covering VNET jails.

Selecting the jail userland version

Most of the time you probably want the userland in your new jail to match that of the host machine. In this case just leave it at CBSD’s suggestion of ‘native’. You can use older versions if you have a specific use case. I’ve been using this to quickly test a port that I maintain on all supported FreeBSD releases. You cannot run a jail with a newer userland than the kernel running on the host machine.

What you can do is run ancient FreeBSD versions in jails – but this is not a good idea in all but very special cases. If you think about doing that for anything but fun, do your fair share of research beforehand. Not only will the standard means of deploying a jail with CBSD not work with FreeBSD older than 9.0, but you will be facing very interesting other changes… (Believe me, I’ve had to save a one man business by taking a crucial internal application – for which the author has long since passed – off the dying hard disk of the FreeBSD 4.11 machine and keep it running on modern hardware…).

Then you need to decide whether you want to have your jail base system writable or read-only. The default setting for baserw is not allowing for writing. Also you should decide on auto starting the jail when the host system booted up or not. The default for astart is yes.

Network interface selection

Select the network interface to use next. For a standard jail you will probably want to simply leave this on ‘auto’, but it doesn’t hurt to define it explicitly, either.

Please note that CBSD also supports VALE(4), netgraph(3) and CBSD VPC. You don’t see these options here because they require additional configuration to be done beforehand.

Architecture selection

In almost all cases you will want to leave the architecture as ‘native’ which means: Same as the host machine. On amd64 you can create an i386 jail if you really want to. Why are there other options listed there as ‘unsupported’ (see screenshot above)? This does not mean that CBSD won’t support such a choice as one could think. If you try to pick one, CBSD will tell you why: Those architectures require emulation on the current platform and a new enough installation of Qemu was not detected!

If you satisfy the requirements and come back, CBSD will allow for creating e.g. a riscv jail on amd64. While this is pretty cool, it’s also clearly advanced usage. I intend to revisit that later in this series but won’t promise anything, yet.

You can enable or disable the VNET feature for your jail next. We’ve touched what this is in the profiles section, and indeed it makes sense to decide upfront and tune the other options accordingly.

Defining the devfs ruleset to use

CBSD wants to know which devfs ruleset to use for the jail. If you don’t know what that means, go with the default for now, but make a note to read more about it. As long as you don’t understand what’s happening there, this will make jails less useful for you. You will hit limitations and probably think something simply cannot be done while it clearly could by using different rules here.

Here’s a very short summary: You know that Unix idea of “everything is a file” and that even devices are represented as files under /dev in the filesystem hierarchy. Well, devfs(5) is what is used to populate /dev with the device nodes for devices that would otherwise only be known to the kernel. There’s interesting stuff in there. Stuff that you might want to keep inaccessible in your jails, especially when you give somebody else root access to those. Be sure to read devfs.rules(5) and peek into /etc/defaults/devfs.rules to see what that ruleset 4 that CBSD suggests, does. For VNET jails you may likely want to use ruleset 5. Define additional rulesets in /etc/devfs.rules if you have special needs.

Create jail from snapshot

On systems that run on ZFS you can select a snapshot as the source for jail data. This is a quick and convenient way to jail something that you have in a dataset on the host machine. If you want the jail data do be encrypted using ZFS encryption, you can opt to do that.

Additional jail options

Next is a list of currently 33 jail options. We’re not going to discuss those here. For almost all of them there’s a description at the bottom line as you move through the menu. Those that don’t have one bear names which make quite clear what they do, with one exception: ‘allow_sysvipc’. If you enable this option, the jail will allow for SysV-style inter-process communication (per jail). You need this for some applications to work (the most popular example may be PostgreSQL).

What follows are some convenience options: You can set the root password in the jail, add a user, configure services and decide whether to do the pkg(8) bootstrap or not.

When you’re happy with your choices, select PROCEED!.

Creating the jail

Create jail immediately?

Now CBSD is asking if it should actually create the jail now. You may be seriously tempted to answer: “Yes, please! I haven’t navigated through all of those menus and selected to proceed instead of cancelling for nothing, you know!”. But there’s a good reason that CBSD makes this last prompt. If you choose NO here, it will just write a jail config file for you which you can edit and e.g. create multiple jails from. So you can use jconstruct-tui both for creating a jail and for composing a template. But of course we’re going to let it create the jail now.

How to acquire the base system?

CBSD would really like to create the jail as asked for, but it can’t. Why? I’ve requested a FreeBSD 13.1 jail (native version on a 13.1 host), yet it has no clean 13.1 userland available to use. And since it cannot create a jail out of the thin air (well, it could but that jail would not be very useful…), it needs to get the base system from somewhere. This is where there are several options and CBSD would like to know which one you prefer:

  • build = load the system source from the net, then compile the base system (takes a long time)
  • extract = use a base.txz dist set somewhere in your host system’s filesystem (copy from an ISO or download before you proceed)
  • populate = create an image of the base system from your host (including the current patch level)
  • repo = download the base.txz from a known location then use that

If you’re creating a jail for a version of FreeBSD that is currently supported, the last option is the easiest. If for some reason you need 10.4 for example, your best option is to manually obtain the base.txz archive, copy it to /tmp and use the second one instead. I would only use the first option if I need to customize the base system (playing with src.conf(5) and its WITHOUT_$SOMETHING directives is time-consuming and can lead to dysfunctional builds – but it’s really cool nevertheless).

Downloading base

I’m going with the repo option here which is also the default. CBSD does a quick speed test of the known mirrors, then picks the fastest one and downloads the compressed tarball. When that’s done, it extracts it and takes various steps to prepare the base system. This only needs to be done once for every release version, of course. So the next 13.1 jail can use the already existing base, but for a 12.4 jail a new base would have to be downloaded and prepared.

Jail has been created

All done, jail has been created! But did it really take 5.5 minutes (see screenshot)? Yes, it did, but that’s mainly due to my bad internet connection. Setting up jails using already fetched and prepared bases takes far less time (about 30 to 60 seconds or so, depending on your hardware).

Managing jails

Ok, so we’ve created a jail. What now? CBSD was nice enough to print out a couple of commands we might want to run next. Note here that it says “VM” a couple of times while it actually means jail. There are places where the opposite happens and it refers to Virtual Machines as “jails”. The reason for this is that CBSD started as just a jails manager and then got more and more support for other virtualization domains. Cleaning up all the cases where those references are incorrect (strictly speaking) and making the output dynamic is a lot of work. It’ll probably happen eventually, but for now get used to it and don’t let this confuse you.

Listing jails and starting the new one

We’re going to run some of the suggested commands in a minute. But before we do, let’s see what services are listening on the host machine:

# sockstat -4l
root     sshd       1578  4  tcp4   *:22                  *:*
root     syslogd    1477  7  udp4   *:514                 *:*

Alright, SSHd and syslogd. Fair enough. Let’s list all the jails on this system next. There should be one now after all, right?

# cbsd jstart
List of offline jail
 0 .. CANCEL
 a .. demo on local

Jail is running now

Okay, should be running now. But it doesn’t hurt to check by listing the jails again:

# cbsd jls
demo   1  demojail.advancebsd.net  /cbsd/jails/demo  On

Yes, it’s on. Now do this to get into the jail:

# cbsd jlogin
List of online jail
 0 .. CANCEL
 a .. demo on local

After selecting the ‘demo’ jail, the prompt changes. We’re in the correct jail:

demojail:/root@[23:04] #

Looks good, but there’s one caveat: FreeBSD prints the kernel version here (you’ve noticed GENERIC, right?). So this info is about the host system! The jail userland is fresh, non-updated FreeBSD 13.1:

# freebsd-version 

But the prompt is correct, we’re in the jail:

# hostname 

Enabling SSHd in the jail and checking on the host

Let’s enable SSHd now:

# sysrc sshd_enable=YES
sshd_enable: NO -> YES

And of course start it:

# service sshd start
Generating RSA host key.
3072 SHA256:nFhZbm6vvaddB0HGLKAUA36LZiYNCtJT38Kn+OfA/18 root@demojail.advancebsd.net (RSA)
Generating ECDSA host key.
256 SHA256:51gPonAdTKmmrC0z1/fGAYdZi/YpkoxVIr6+FnQe1tM root@demojail.advancebsd.net (ECDSA)
Generating ED25519 host key.
256 SHA256:PGoUUapX8HX1bHUzehVA13BOhEpgYmfLP9m3vf2JHbc root@demojail.advancebsd.net (ED25519)
Performing sanity check on sshd configuration.
Starting sshd.

How do you leave a jail? Simply by ending your shell session:

# exit

So we’re back on the host system. If we look at the services listening, we should see a difference:

# sockstat -4l
root     sshd       10978 3  tcp4           *:*
root     sshd       1563  4  tcp4   *:22                  *:*
root     syslogd    1474  7  udp4   *:514                 *:*

And there we are: There’s a second SSHd running and bound only to the jail IP. We could now create a user in the jail and try connecting from outside instead of going via jlogin from the host machine, but the NATting is not in place yet!

Stopping the jail

This is a good point to end this article, though. We’ve covered a lot of ground again. It wouldn’t be quite complete without at least stopping the jail again first, though. So let’s do this final step and then call it a day:

# cbsd jstop
Stoping jail: demo, parallel timeout=5
jstop done in 2 seconds

Checking the jails list again real quick:

cbsd jls
demo   0  demojail.advancebsd.net  /cbsd/jails/demo  Off

Yes, jail is down. That means there should again only one SSH daemon be running on the system now, shouldn’t it?

# sockstat -4l
root     sshd       1563  4  tcp4   *:22                  *:*
root     syslogd    1474  7  udp4   *:514                 *:*

Indeed: The second one that was running inside the jail is gone.

What’s next?

After covering the bare basics, next time we’ll have to look a bit more into managing jails.

2 thoughts on “Exploring the CBSD virtual environment management framework – part 3: Jails (I)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.