FreeBSD jails (2/2): 4.11 sentenced to jail

Ok, after introducing FreeBSD’s jails and jail frameworks in the previous post, it’s time to actually put that 4.11 system that was built in February behind bars. Before turning towards that special case, though, we’ll have a look at how to use iocage (one of the jail management frameworks).

Creating and starting an 11.0 jail

We’ve built iocage from ports last time, so let’s just jump right into it and try out one of the most important commands:

# iocage list

Iocage initialized

Not too surprisingly, the jail list is empty. But the important thing here is that iocage automatically created its infrastructure when run for the first time. This is generally fine; if you happen to have more than one ZFS pool, though, you may want to manually activate one for iocage usage.

The next thing to do is to fetch the distribution tarballs for the FreeBSD version that shall populate the jails you want to create:

# iocage fetch -r 11.0-RELEASE

Fetching the 11.0 release

This process will also update the kernel and userland to the latest available version of the selected release. I chose 11.0 because that’s the latest right now and obviously 4.11 is not available!

You can actually skip fetching the release as iocage will do it automatically if you create the jail, but doing it beforehand doesn’t hurt. Now let’s create our first jail:

# iocage create tag=fbsd11 ip4_addr="em0|" -r 11.0-RELEASE

Jail ‘fbsd11’ created!

Each jail has a UUID. This one’s UUID begins with 99a9941a. You can interact with jails by using that UUID. But if you’ve got a lot of jails it would be pretty hard to keep track of which one is which. For that reason you can assign a tag to a jail that you can use to identify it. We’ve effectively given our jail a name: fbsd11. In general it’s a good idea to name it in a way that helps you (or your co-workers) to understand what the jail is used for. You may choose something like “postgres-jail” that includes the technology used, “loadbalancer” to describe its function or whatever fits your needs.

We’ve also assigned the jail an ipv4 address, in this case, noted the subnet in CIDR notation and specified which NIC to use it on. Listing jails shows all that information at a glance.

The jail is ready to be started:

# iocage start fbsd11

Starting the jail

As you can see here from the ifconfig output, the ip address was not yet assigned to the NIC. According to iocage the jail has been started successfully. Let’s take a look at it:

# iocage list

The jail is running

The state for our jail has changed to “up”. Also the ip has been automatically assigned to our network interface when iocage started the jail. When the jail is stopped, it is also automatically removed again.

Using the jail

Iocage offers some means to interact jails without actually “entering” them, like e.g. installing packages, etc. The easiest way to use a jail however, is to get a shell prompt inside the jail (think entering a chroot environment). That’s what we’re going to do next:

# iocage console fbsd11

Entering the jail console

Looks pretty much like we just logged into a FreeBSD system, huh? Well, we basically did just that. With the big difference of not being a full OS (which would be the case with a VM for example) but a jail with just a FreeBSD 11.0 userland (that is running on the host machine’s kernel).

Ok, I bootstrapped pkg and then installed nginx:

Installing the nginx webserver

This also proves that the network is working since we’re receiving the packages to install from. There’s also nothing special in doing this, just use pkg the way you’re used to. For the most part it’s behaving just like a regular FreeBSD system. There are a few exceptions. One of the most notable ones is the fact that by default you won’t be able to use the ping command. You can enable jails to use ping using a sysctl but this affects the jails security and therefore is disabled by default.

Let’s start nginx next:

Nginx is running

Again it’s nothing special. Nginx is running and bound to port 80. Does it do what you’d expect it to?

Testing and destroying the jail

Nginx’s default page

Yes, the webserver is happily serving the default page when accessed from other computers on the LAN! It doesn’t make much of a difference from running nginx on the host system – unless somebody manages to use an exploit. Nginx is privilege separated and thus probably not a good example. But think about a case where an intruder manages to get into your system and get root privileges. That’s pretty bad, yes. But if the application was running in a jail, the damage is at least contained and the other jails as well as the host system are unaffected. (That and it’s much easier to tear that jail down and create a new one compared to having to reinstall the whole host system and setup several applications again!)

Alright. Let’s take a quick look at where the jail data is kept:

Taking a look at iocage’s directory structure

Iocage created the /iocage dataset. The most important path is /iocage/jail. All the jails are in there (named after their UUID). To make accessing them easier, there’s /iocage/tags: Here you’ll find symlinks named after the jail’s tags which are pointing to the actual directories with the data.

Each jail consists of a configuration file, an fstab and a “root” directory (which holds all the files).

This jail has served its purpose and is no longer needed. So let’s get rid of it! First exit the jail like you would exit a chroot (e.g. CTRL-D out of it) before you proceed:

# iocage stop fbsd11
# iocage destroy fbsd11

Stopping and destroying the jail

You cannot destroy a running jail. Therefor it must be stopped first and can then be destroyed.

Creating the 4.11 jail

That has been pretty easy and straight forward, right? And here’s the good news: Jailing the 4.11 system isn’t that difficult, either. I created a tarball of the complete filesystem on the FreeBSD 4.11 server, xz compressed it and scp’d that onto the jailhost. Now we’ll create another 11.0 jail, delete the filesystem and extract the old OS data into the same place:

# iocage create tag=fbsd4.11 ip4_addr="em0|" -r 11.0-RELEASE
# cd /iocage/tags/fbsd4.11/root
# rm -rf * .*
# tar xvJf /path/to/tarball.xz

Creating a new jail and putting the 4.11 OS into it

The old device nodes won’t work on a current kernel and have to be removed. The same goes for the old proc filesystem. We also don’t need any kernels or kernel modules. And /boot is obsolete in a jail, too. So it’s a good idea to get rid of all that:

# rm dev/*
# rm kernel*
# rm -rf boot
# rm -rf modules*
# rm -rf proc/*

Deleting unneeded parts of 4.11

The old userland and the additional data remains. Let’s try to start the jail now:

# iocage stop fbsd11

Starting the 4.11 jail

Uhh… This doesn’t look good. Iocage tries to do things that won’t work with a 4.11 userland (anything older than 9.3 is unsupported with iocage, anyways!). But fortunately that looks worse than it actually is:

It errors out… but it’s running!

Despite those errors the jail is up and running! Can we get console access?

Inside the 4.11 jail

# iocage console fbsd4.11

Entering the console works, too!

Yes, we can! Let’s make 4.11 great again! Time to install a webserver in this jail, too. Since we still have Pkgsrc in place, that’s an easy thing to do:

# cd /usr/pkgsrc/16/www/nginx
# bmake install clean clean-depends

Building nginx on 4.11 through Pkgsrc

I decided to modify nginx’s default starting page so it looks a little different from the stock one. Nginx can be started simply by executing the daemon. However all the system utilities regarding process management don’t work! And sockstat is not working either.

Starting nginx, but tools like ps don’t work!

Does nginx actually run? There’s an easy way to test that. Let’s just open up a browser on another pc and see if we can get the modified webpage.

Nginx is able to serve the webpage from the 4.11 jail!

It works! Yes, it was actually that easy. Jails are not necessarily a complicated thing at all (they can be more complicated than that of course, if you’re doing more advanced stuff). But… No ps, no top, no kill… How are you supposed to work with a 4.11 jail?

Peter Wemm suggested using statically compiled binaries from the host system like those in /rescue. And that’s actually a great idea! If you copy /rescue over into the jail, it at least provides some tools to work with.

Thanks to /rescue process management is possible

Things like top would be nice, too, of course. But to be honest, I don’t know how to statically build them. If anybody knows, please comment and share your wisdom!

More about jails?

This article merely gave a very basic introduction to the topic (plus a little “stunt”). You can do a lot more with jails. I might write about it again in the future but I haven’t decided, yet (feel free to comment and tell me what you’re interested in!). But I wanted to at least give you an idea what other things there are to explore.

If you have a server with only one ip you can still use jails. The solution to this is to give the jails internal network addresses (like 192.168.x.x) and then use Pf to do network address translation (NAT). It took me a bit to figure out how this works (there are things on the net that are better documented) but in the end it’s not that complicated.

With VIMAGE/VNET you can give your jails their own virtual network stack. That way you can even use a firewall inside the jail and assign multiple ip addresses!

Jails support quotas of various kinds. You can limit the amount of CPU that one jail may occupy, limit file system IO, etc. Lots of possibilities with this.

And of course iocage has many nice features. You can e.g. create jail templates with some software already installed and quickly create jails from those. There are multiple jail types like e.g. thin-provisioned jails. You can clone and snapshot jail. And you can have jails started automatically when your server boots.

In the previous article I said “think containers” when you’re completely new to jails. Now you’ve seen a bit of what they can do and have a better idea of what they are. So stop “thinking containers” now. These are two things that are somewhat related but they are not the same thing. Linux’s containers can do things that jails were never designed for (e.g. sharing namespaces). But jails have one huge advantage over containers: While containers are only a means for logical separation, jails were designed for security right from the start. Sure, people are working on making Linux’s containers more secure. But if there’s one thing that we should have learned by now, it is this: Retrofitting security into something is not really going to work… And jails are pretty flexible, too. You might not have missed them when you didn’t know that they exist. But get used to using them once and you’ll think differently about a lot of things and make use of them for more and more things. Jails are a great feature of FreeBSD.

3 thoughts on “FreeBSD jails (2/2): 4.11 sentenced to jail

  1. > Things like top would be nice, too, of course. But to be honest, I don’t know how to statically build them. If anybody knows, please comment and share your wisdom!

    On host system in /usr/src/rescue/rescue/Makefile search line ‘CRUNCH_PROGS_usr.bin+=’ (196 in FreeBSD 11.1) and add top:
    CRUNCH_PROGS_usr.bin= head mt sed tail tee

    CRUNCH_PROGS_usr.bin+= gzip
    CRUNCH_PROGS_usr.bin+= top systat

    Read /usr/src/rescue/README:
    cd /usr/src/rescue && make obj && make && make install

    Copy /rescue/top and /rescue/systat to /usr/bin/ into jail.

    It works for me.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.