Back and forth: Linux and *BSD

This is kind of the post that I wanted to write much earlier this year. After running a Linux-only environment at home for years, I had become less and less happy with the general direction things seem to be heading. I had run FreeBSD and OpenBSD on real hardware (old laptops) and several versions of PC-BSD in VirtualBox over the years. In January I decided to step forward and install PC-BSD (10.2) on my primary computer for daily usage. It remained a short episode – and this post will describe why. When TrueOS was released to the public I decided to try out that right away. But that will be another post.

Initial contact

I cannot remember when I first read about the BSDs. That must have been many years ago when I became interested in reading a bit about UNIX. I remember beastie and puffy and I remember that I failed installing a system in a VM because it was somehow too complicated. It likely was OpenBSD and the chance is quite high that I quit during the partitioning which probably was way over my head at that time.

While I never lost interest in it (Unix fascinated me) I decided to “learn Linux first” as that was the system I had chosen to run my computers with. As the Linux world was big enough for years (trying out the various desktops, doing a lot of distro hopping, …) I touched *BSD only rarely. Basically it was limited to installing PC-BSD in a VM when I found out that a new version was released. It seemed to be nice but I didn’t see any benefit over my Linux systems and so I stuck with that.

After studying something entirely different, I had made the decision to break up and get into the IT instead, even though was I well beyond the age that you usually start an apprenticeship. In my country that means that you apply to a company to work as an apprentice there half of the week and go to school the other days. Being somewhat of a Linux nerd I had only applied to companies that I knew weren’t using Windows – I had left that mess and was determined to avoid it in the future as far as possible. In the end I signed a contract of apprenticeship with a hosting company, moved into the area and started learning Linux a lot deeper than I had before. And… I came in contact with FreeBSD.

Being a hosting company that had been founded in the nineties, it had of course started on FreeBSD. Even though the focus of the company shifted to Linux years ago, there still were about 100 servers running FreeBSD. My colleagues generally disliked those servers – simply because they were different. And our CIO declared that he hated them and would love to get rid of them as FreeBSD was totally obsolete these days. If it hadn’t been for our boss to have a soft spot for them (as that had been what he started with and also what he had come to know best over the years) there definitely would have been far less FreeBSD servers around.

Digging into FreeBSD

Now for whatever reason I do have a heart for underdogs and so I begun to be interested in those odd systems quite a bit. Nobody wanted to touch those dinosaurs if he didn’t really have to. However somebody had to take care of them anyways, right? They were production servers after all! I volunteered. There were moments where I kind of regretted this decision but now in hindsight it was an excellent choice. I’ve learned a ton of little things that made me understand *nix and even the IT in general quite a bit better compared to what I would know now if I had followed the straight Linux path.

I also found out that only very few things that the colleagues hated about our FreeBSD boxes were things to actually blame FreeBSD for. By far the biggest problem was that they simply had been neglected for like a decade? Our Linux systems used configuration management, the FreeBSDs were still managed by hand (!). We had some sophisticated tooling on Linux, on the BSD boxes there were crude old scripts to (kind of) do the same job. Those systems were not consistent at all; some at least had sudo others made you use su if you needed to use privileged commands… Things like that. A lot of things like that. So it wasn’t exactly a miracle that the BSDs were not held in very high regard.

As I said, I didn’t really see any real advantage of BSD before. Linux even seemed to be easier! Think network interfaces for example: “eth + number” is easier than “abbreviation of interface driver + number”. But Linux has since moved to “enp0s3” and the like… And when you think again, it does make a lot of sense to see what driver an interface uses from the name. Anyways: I begun to like that OS! FreeBSD’s ports framework was really great and I realized the beauty of rc.config (Arch Linux did away with their central config file to get systemd. What a great exchange… – not!). Also I liked the idea of a base system quite a bit and /rescue was just genius. Would my colleagues lose their contempt for our BSD servers if they were configured properly? I thought (and still think) so.

My apprenticeship was nearing its end and I had to choose a topic for the final project work. I was advised to NOT do something Linux related because the examiners… *cough* lacked experience in that field (in the past an apprentice even failed because they have no idea what they are doing. He went before court and it was decided in his favor. A re-examination by people who knew Linux got him an A!). Now things like that make me angry and calls upon the rebel in me. I handed in a FreeBSD topic (evaluating Puppet, Chef, SaltStack and Ansible for orchestration and configuration management of a medium-sized FreeBSD server landscape).

So for servers I was already sold. But could *BSD compete on the desktop, too? I built two test systems and was rather happy with them. However I wanted to try out a BSD system optimized for desktop usage. Enter PC-BSD.

Working with PC-BSD

I was called nuts for making that switch just days before the final presentation of the written project work (“you need to pass this – your entire career depends on it!!”). But I didn’t want to do a presentation on a FreeBSD topic using a Linux machine! Well, in fact I had been too optimistic as the installation turned out to be… rather problematic due to a lot of bad surprises. To be fair: Most of them weren’t PC-BSD’s fault at all. The BIOS mode on my computer is broken in it not supporting booting off GPT partitions in non-UEFI mode. This lead to my drives disappearing after installation – and myself wondering if my classmates were right… Never change a running system! Especially not if you’re pressed for time!

After I found out what the problem was, installing to MBR was an easy thing to do. I still needed every single night that I had left but I got everything to work to at least the level that allowed me to hold my presentation. Another thing was that I had enabled deduplication on my ZFS pool. “24 gigs of memory should be enough to use that feature!”, I thought. Nobody had told me that it slows down file deletion so much that deleting about 2 GB of data meant to go and do something else while ZFS was doing its thing. Even worse: The system was virtually unresponsive while doing that so you could forget browsing the web or something like that in the meantime. But truth be told this was my own mistake due to my very own ignorance about ZFS and I can hardly blame PC-BSD for it.

I kept PC-BSD on my laptop for about 1.5 month before I needed to return to Linux – and I would in fact even have returned earlier had I had the time to reinstall. While some issues with PC-BSD vexed me, too, I could have lived with most of them. But my wife complained all the time and that of course meant the end for my PC-BSD journey.

So what were (some) of the issues with it? My wife mostly uses the PC to check email when our children are occupied with something for a moment. For her the very long boot time was extremely annoying. And really it took multiple times as long as the Linux system before (and that was still one with Upstart!). Keeping one user logged in and changing to another user quickly wasn’t possible – which meant that I had to shut down my multiple virtual machines and log out completely if my wife just wanted to quickly check mail or something. Not cool. Things like that.

And then there were a few things that annoyed me. It drew power from the battery much, much faster than the previous Linux system. When watching a video, the screen saver kept interrupting it. Firefox had strange issues from time to time and liked to crash. Working with EXT4 formatted disks was a pain. And so on and so forth.

Of course there were good parts, too. I had a real FreeBSD system at my hands with access to ports. Two firewalls (that are nothing like the mess that is netfilter/iptables!) to choose from. Excellent documentation. Nice helper tools (like the automounter, wifi manager, disk manager, etc.). Several supported desktops to choose from. And of course the well thought-out update system that I liked a lot. Thinking about it, there are a lot of good parts actually. Unfortunately even a ton of things nice to have have a hard time covering things conceived as no-gos. That’s life.

I had intended to update to 10.3 and then write a complete blog post about PC-BSD. My wife didn’t like the idea much, though. In addition to that I had little spare time and no alternative spare hardware, so there wasn’t a chance for me to actually do that.

Interlude: Linux

So it was back to Linux. With systemd this time. I’m not exactly friends with that omnivoristic set of tools that annoyed me perhaps just not enough to switch the system over to runnit or openrc. Other than that life was good again (as my wife was happy and I could do my work). But there was one thing in the short period of time with PC-BSD that had changed everything: I had caught the bug with ZFS!

Fourtunately there’s ZFSonLinux, right? So I installed that and created a pool to use for my data. In general that worked but it’s a bit more hassle to set up compared to FreeBSD where you basically get it for free without having to do anything special! If you don’t want to compile all packages related to ZFS yourself for each new kernel, there’s a third-party package repository for Arch. ZFS is not in the official ones. At some point the names of the packages changed and the update failed. I didn’t find anything about that and had to figure out myself what happened.

After another kernel and ZFS update that I did in the morning succeed. But when I came home, my wife told me that when she logged in, she was logged out again almost instantly. I booted the computer and logged in – the same thing happened. What was that? No error message, no nothing. The system simply dropped me back at the login manager… So I switched to text mode to take a look at what might be wrong with the system. Long story short: My pool “homepool” which held all user’s home directories was not available! And worse: zpool import said that there were no pools available for import… With the update, ZFS had stopped working! That hit me in the wrong moment whan I had very little time and so I had to downgrade as the quickest solution.

In the end I chose to compile the “solaris porting layer” and the other packages myself. This was not so bad actually but knowing that on FreeBSD I’d have access to ZFS provided by the operating system without having to do anything (and that nobody was going to break it without it probably being fixed again in no time) vexed me. Of course there were other things, too, and using FreeBSD on other boxes, I wanted it back on my main desktop machine as well.

What’s next?

I installed TrueOS and used it for over three months. The next post will be a critical writeup about TrueOS.

Bacula on FreeBSD (pt. 5): A day at the pool

Edit: I put up the original post only for a short time before I decided that I wasn’t really happy with it at all. So I took it down again to rework it a bit. By now I’ve more or less rewritten it completely. I hope that this version will be more useful.

This is part five of my Bacula tutorial. The first part covered some basics as well as installing Bacula and starting all three daemons. Part two dealt with allowing all components of Bacula to interact with each other locally, debugging config problems and doing a first test backup was.

In part three the configuration was cleaned up and split into smaller parts, the first self-created resources (fileset, device and storage) were added and a backup job customized using the bconsole. Part four detailed restoring files from backup and discussed jobs as well as volumes, labels and pools.

Part five will show how to create a new storage pool. We’ll use memory disks so that we can simulate partitions. Then we’ll talk about jobs again and have a closer look at volumes in our pool and some of the states they can have. Finally we’ll briefly touch upon the topic of recycling volumes. You’ll need to know all that (and actually some more) prior to planning your real pool(s).

The fifth part continues where the third one left off. During this tutorial series we use VirtualBox VMs managed by Vagrant to make things as simple as possible. If you don’t know how to use them, have a look here and here for a detailed introduction into using Vagrant and on how to prepare a VM template with FreeBSD that is used in the Bacula tutorial.

Preparations for a storage pool

Load the latest snapshot, enter the VM and switch to root:

% cd ~/vagrant/backuphost
% vagrant snapshot restore tut_4_end
% vagrant ssh
% sudo su -

Our VM has a simple partition scheme where essentially everything is put in one partition. A volume that provides storage backed by a file will keep growing until the disk is full. To enforce restrictions we have to create a pool. Also our VM certainly does not have a tape drive or anything like that. Still I’d like to simulate filling a medium! How can we do that? Probably by creating virtual disk-backed storage for which we can set a fixed size.

This time we’ll backup a directory with a bit more data in it. I suggest /usr/bin. Let’s see how big it is (mind this size! Especially if you’re using a version newer than the 11.0 release that I’m using here the size may be different):

# du -sh /usr/bin

143M /usr/bin

Ok, we’re going to create a sparse file of 300 MB in size next (so that at a bit more than two full backups would fit in there) as well as a directory for the mountpoint:

# truncate -s 300M /var/backup/ufs0
# mkdir /mnt/storage0

Now we need to make the storage accessible as a device rather than just a file. FreeBSD calls these devices memory disks because most of the time you’ll want such a device to be backed by RAM (to benefit from its speed), but file-backed storage is also possible. As a first step we need to create the device and, as a second one, put a filesystem on it. Then in step three we can mount it on a directory to make it available on the machine’s global filesystem hierarchy. Fortunately FreeBSD is pretty good in doing memory disks and comes with a command that can do all three steps at once:

# mdmfs -F /var/backup/ufs0 -S -n -p 700 -w bacula:bacula md0 /mnt/storage0

Let’s check if that worked:

# mount

See the mount? Looks like we have our file backed storage in place.

Configuration changes

Now we can create a new fileset to back up /usr/bin:

# cd /usr/local/etc/bacula
# vi includes/dir_fileset.conf

Add the following lines at the end of the file:

FileSet {
Name = "usr-bin"
Include {
Options {
signature = MD5
}
File = /usr/bin
}
}

Save the file and exit. Of course we need to tell the sd where to store the backups – we need to create another device:

# vi includes/sd_device.conf

Add this device resource at the end of the file:

Device {
Name = Stor0-Test
Media Type = Test
Archive Device = /mnt/storage0
LabelMedia = yes;
Random Access = Yes;
AutomaticMount = yes;
RemovableMedia = no;
AlwaysOpen = no;
Maximum Concurrent Jobs = 5
}

And the director needs to know about this storage as well:

# vi includes/dir_storage.conf

Add this block at the end of the file:

Storage {
Name = Test
Address = localhost
SDPort = 9103
Password = "sdPASSWORD"
Device = Stor0-Test
Media Type = Test
Maximum Concurrent Jobs = 10
}

We’ve done all this before but the next step is new. We’re going to create a pool:

# vi includes/dir_pool.conf

Again add the following lines to the file:

Pool {
Name = Testpool
Pool Type = Backup
Recycle = no
Maximum Volume Bytes = 90M
}

Just set recycle to “no” for now – you’ll see what that does in a minute. We also want to force a maximum volume size of less than what the backup is worth of data so that two volumes will be needed for one full backup. And to avoid having to use “mod” at the bconsole all the time, we’ll add a new job for convenience:

# vi includes/dir_job.conf

Here’s the job resource to put at the end of the file:

Job {
Name = "TestJob"
JobDefs = "DefaultJob"
Level = Full
FileSet = usr-bin
Storage = Test
Pool = Testpool
}

Jobs

Ok. Our basic preparations are done. Let’s restart the daemons and then try and see what happens if we run our new backup job:

# service bacula-dir restart
# service bacula-sd restart
# bconsole
* run
4
yes
* mes

[…]
29-Oct 12:48 backuphost.local-sd JobId 4: Job Testjob.2016-10-31_12.48.33_03 is waiting. Cannot find any appendable volumes.
Please use the “label” command to create a new Volume for:
[…]

The job cannot start because there are no volumes that Bacula could use. We could now create volumes using the label command as we’ve done before. But we configured our device with the LabelMedia directive set to “yes”! It should create volumes automatically as needed! Time to look at the configuration again. But let’s first take the chance to cancel the currently pending job.

We already know the JobId from the message above. But let’s pretend we didn’t know. We’ll ask the director for an overview of jobs (both past and present):

* list jobs

There we have a simple table. Let’s see what information it holds. First we have a unique JobId for each job. The second column holds the name of the backup job – this will usually be the name of the client or RestoreFiles for a restore job. But as you can see in our case, “TestJob” will work as well (however you really should stick to names that hint which client they belong to in a production environment as things would get pretty confusing really fast). StartTime is self-explanatory. Type is “B” for backup and “R” for restore in our example. We’ve only run full backup level jobs so far. JobFiles and JobBytes is self-explanatory again. And in case of JobStatus, a “T” means terminated, an “R” running, “A” for aborted, etc. Now we’ll cancel the job that is waiting for a new volume:

* cancel 4
yes
* mes

[…]
Backup Canceled
[…]

So we’ve canceled the job. Let’s exit the console now and take a look at the pool configuration again:

* exit
# vi includes/dir_pool.conf

Our pool resource is missing a directive that specifies how the label names are to be composed. We should add that one line real quick:

Label Format = "Test-"

Now we need to restart the dir and invoke the bconsole again.

# service bacula-dir restart
# bconsole

Turn up the volume(s)!

Then we can take a look at the volumes that we have so far:

* list volumes

[…]
Pool: Testpool
No results to list.
[…]

The pool Testpool is empty right now. Take a look at the pools used for our previous jobs and get an idea of what they look like. Now let’s run the test job again and see what happens:

* run
4
yes
* mes

[…]
Labeled new Volume “Test-0003” on file device “Stor0-Test” (/mnt/storage0).
[…]

So auto labeling obviously works. This is one huge benefit of using a pool! But there are others like limiting the volume sizes and many more. Did the backup complete successfully by now?

* mes

[…]
Backup OK
[…]

It did. Time to look at the volumes again; look for the column VolStatus:

* list volumes

[…]
Full
Append
[…]

The first volume has a VolStatus of Full and the second one is Append which means that more backup data can be written to it. We’ll do that by simply running our test job again:

* run
4
yes
* mes

[…]
WARNING: device is full! Please add more disk space then …

Please mount append Volume “Test-0005” or label a new one for:
[…]

We’ve created a really small ramdisk for /mnt/storage0 and it is full before the second full backup could be completed. But how’s that? There’s 300 MB of space and the fileset backs up less that 150 MB! Is there so much overhead? No, not really. What has happened here is that we restricted the pool to volumes of 90 MB each. The three volumes occupy 270 MB – and while there’s 30 MB more left on the pool, that’s too little space to create another volume on it! So what do we do now? First have a look at the volume list again:

* list volumes

[…]
Full
Full
Full
[…]

Coming full cycle

All of them are listed as Full. But there’s some more interesting info there. Notice the Recycle column? We’ve forbidden Bacula to recycle old volumes when we defined the pool resource. We can fix that, right? Let’s exit the bconsole and edit the configuration file:

* exit
# vi includes/dir_pool.conf

Change the respective line to:

Recycle = yes

Then save the file and exit the editor. The configuration changed and so the dir needs to reinitialize. To do so we restart it.

# service bacula-dir restart

Seems like it’s not responding? Hit CTRL+C to cancel. Let’s stop and start it instead:

# service bacula-dir stop
# service bacula-dir start

That worked. Now we enter the bconsole again and take a look at the volumes:

# bconsole
* list volumes

Huh? That configuration change didn’t work! The recycling flag is still set to 0. Why? There is an easy answer to that: Because this value is not read from the configuration! It comes from the catalog. The configuration setting is applied at the moment a new value is created. Once the volume exists, the configuration setting is irrelevant. Of course we are not out of luck here. We can modify the flag using the bconsole (and yes, that asterisk right before the MediaId (the three in this case) is NOT a prompt symbol; type it in!):

* update
1
7
4
*3
* yes
18

Let’s see if that worked:

* list volumes

It did! So will Bacula now reuse the old volume and overwrite all data on it? No it won’t. Bacula knows that there’s data on it because it keeps track of all that in the catalog. And it tries to preserve that data even though the volume allows recycling.

Purging a volume

However we can tell Bacula to get rid of the catalog data that references this volume. To do so, we purge job and volume information for that volume from the catalog:

purge [purge jobs volume]
3
4
*3
* list volumes

[…]
Purged
Full
Full
[…]

See how the VolStatus changed? It’s purged and recycling is enabled. That means that Bacula will reuse the volume. But will our job resume automatically? Let’s take a look at it:

* list jobs

Oh no! What’s that JobStatus? It’s “f” for failed! What happened? Well, we stopped the director, remember? That killed the running job! So to try out recycling we need to run another backup job. Let’s start the job now:

* run
4
yes

Take a look at the volumes again:

[…]
Recycle
Full
Full
[…]

The VolStatus changed to Recycle and Bacula will reuse the old volume. In theory we’d have to purge one more volume for our backup to succeed. But since this is just an example job to show off some important things, we’re actually done at this point. But for today’s tasks we’re done now.

Save the current status for later:

* exit
# shutdown -p now
% vagrant status
% vagrant snapshot save tut_5_end

Intermission

You now know the basics of pool creation and some of the features that come with it. You’ve also purged and recycled a volume and should have a better understanding of how Bacula works in general. There’s a lot more to pools however and the next post in this series should probably go into retention periods and discuss a topic that we’ve only touched upon so far: The Catalog.

However this post concludes my „Bacula October“ and I’ll end this tutorial series here. It takes a lot of effort and time to write these posts and while I hope that this is of any use to somebody, I have no idea whether it is or not. For that reason I might or might not take this topic up again in the future. I had planned to simulate multiple backup clients with vagrant, do encrypted backups and so on. But now I’m looking forward to write about something else again! Of course feel free to comment on any of the parts if you liked (or want to tell my why you didn’t like) this tutorial.

Bacula on FreeBSD (pt. 4): Jobs, volumes, pools & a restore

This is part four of my Bacula tutorial. The first part covered some basics as well as installing Bacula and starting the three daemons that it consists of. Part two dealt with allowing all components of Bacula to interact with each other locally, debugging config problems and doing a first test backup. In part three the configuration was cleaned up and split into smaller parts, the first self-created resources (fileset, device and storage) were added and a backup job customized using the bconsole.

Part four will discuss jobs, and show how to do a restore. We will change the default settings for the backup job so that it’s no longer necessary to modify it using the bconsole. Also volumes, labels and pools will discussed.

The fourth part continues where the third one left off. During this tutorial series we use VirtualBox VMs managed by Vagrant to make things as simple as possible. If you don’t know how to use them, have a look here and here for a detailed introduction into using Vagrant and on how to prepare a VM template with FreeBSD that is used in the Bacula tutorial.

Jobs

We’ve already seen and used backup jobs. There are also jobs for different actions like restore (and some others). But what is a job? It basically is your way of telling Bacula what to do and how to do it. The job type or action defines what Bacula should do in the first place. Back up data? Restore it? Verify (compare) data? The client determines who to operate on; it answers the question: Which host to back up data from or restore data to?

Then there’s the fileset which in case of a backup determines which files should be included and the pool that determines where to store the data. And finally there’s the schedule that determines when a job is run. So far we have only started jobs manually using the bconsole – but that’s certainly not what you have in mind for your backup solution (or maybe for simple backups it is. But in that case using Bacula for your backups is most likely overkill and you might want to look for a simpler backup utility)!

When we did our second backup, we changed a lot of settings using the bconsole. Now let’s modify the configuration instead so that those will be the new defaults for the backup job. Of course we’ll first have Vagrant spin up the VM, SSH into it and so on (you now the score by now):

% cd ~/vagrant/backuphost
% vagrant snapshot restore tut_3_end
% vagrant ssh
% sudo su -
# cd /usr/local/etc/bacula/

Then we can edit the job defaults (if you haven’t read the previous part(s) and wonder why you don’t have that file or even the directory – that’s because we’ve split the configuration for better readability!):

# vi includes/dir_job.conf

The topmost resource should be the JobDefs one that has the name DefaultJob. This is a kind of template for all jobs which only overwrite directives that differ from the default one and just use the rest as set here. Change FileSet to etc, Storage to File3 and Pool to Default. Save the file and exit the editor.

Now restart the director and prepare to run the backup job again using the bconsole:

# service bacula-dir restart
# bconsole
* run

Automatically selected Catalog: MyCatalog
Using Catalog “MyCatalog”
A job name must be specified.
The defined Job resources are:
1: Backuphost.local
2: BackupCatalog
3: RestoreFiles
Select Job resource (1-3):

Choose 1:

Run Backup job
JobName: Backuphost.local
Level: Incremental
Client: backuphost.local-fd
FileSet: etc
Pool: Default (From Job resource)
Storage: File3 (From Job resource)
When: 2016-09-23 23:48:01
Priority: 10
OK to run? (yes/mod/no):

That’s looking like it should. No more need to use mod multiple times! Type no now as we don’t actually need to do another backup at this time.

Restoring files

Instead we’ll be doing a restore next. Issuing the command to initiate a restore job leads to a long list of choices:

* restore
[…]
To select the JobIds, you have the following choices:
1: List last 20 Jobs run
2: List Jobs where a given File is saved
3: Enter list of comma separated JobIds to select
4: Enter SQL list command
5: Select the most recent backup for a client
6: Select backup for a client before a specified time
7: Enter a list of files to restore
8: Enter a list of files to restore before a specified time
9: Find the JobIds of the most recent backup for a client
10: Find the JobIds for a backup for a client before a specified time
11: Enter a list of directories to restore for found JobIds
12: Select full restore to a specified Job date
13: Cancel
Select item: (1-13):

Pick option 5 – the most frequent one that I use, BTW:

Defined Clients:
1: backuphost.local-fd
2: fbsd-template.local-fd
Select the Client (1-2):

Huh? Where’s that fbsd-template.local client coming from? Haven’t we removed it from the configuration completely? Yes, we have. However we did our very first backup when this was still the hostname of the virtual machine and the catalog remembers that it holds a backup for that client! Ignore that for now and select 1:

[…]
424 files inserted into the tree.

You are now entering file selection mode where you add (mark) and
remove (unmark) files to be restored. No files are initially added, unless
you used the “all” keyword on the command line.
Enter “done” to leave this mode.

cwd is: /

Notice that the prompt symbol changed ($)? You’re in a virtual shell now from which you can navigate through a filesystem rebuilt from the files contained in the selected backup. It is fairly limited, however. The most obvious limitation is that it does not provide auto-completion. You’ll have to live with that. And of course it does only provide a basic set of commands that allow you to change paths, list files, etc.

Bacula said that we’re in /. Let’s see what we have there:

$ ls
etc/
usr/

Ok, so obviously our filesystem consists of a subset of both /etc and /usr (subset because we’ve excluded /etc/caspar from /etc and only included /usr/local/etc and not the whole /usr, remember?).

Let’s see what is in /usr/local/etc, shall we:

ls /usr/local/etc

Nothing? Ouch. Does that mean that the backup is broken for whatever reason? No, in fact everything is fine. The problem here is that even this rather simple command is too advanced for Bacula! You want to see the contents of some directory? Go there and have a look again:

$ cd /usr/local/etc
cwd is: /usr/local/etc/

$ ls
X11/
bacula/
bash_completion.d/
drirc
man.d/
pam.d/
periodic/
pkg.conf
pkg.conf.sample
rc.d/
sudoers
sudoers.d/
sudoers.sample
xdg/

There you go. It’s all there. Let’s mark the sudoers file so that it’ll be added to the restore job (you can also use the mark command, but add is shorter!):

$ add sudoers
1 file marked.

Ok, that worked. Just bear in mind that you always have to enter the directory first before you can mark (or view) any files. Even if you know where in the filesystem something is, Bacula can’t cope with anything more complicated than the very basic way of doing things.

Now let’s change to /etc:

$ cd /etc
cwd is: /etc/

I won’t show an ls here since that’d be too much output. But do it yourself and see if /etc/casper was really left out from the backup. Alright. Now let’s assume we want to restore csh.cshrc, csh.login and csh.logout as well. Thankfully Bacula’s virtual shell does support globbing (wildcard expansion):

$ add csh*
3 files marked.

After selecting a bunch of files, let’s tell Bacula that we’ve finished adding files:

$ done
[…]
4 files selected to be restored.

Run Restore job
JobName: RestoreFiles
Bootstrap: /var/db/bacula/backuphost.local-dir.restore.1.bsr
Where: /tmp/bacula-restores
Replace: Always
FileSet: Full Set
Backup Client: backuphost.local-fd
Restore Client: backuphost.local-fd
Storage: File3
When: 2016-09-25 08:02:20
Catalog: MyCatalog
Priority: 10
Plugin Options:
OK to run? (yes/mod/no):

Bacula has prepared a restore job and shows us a summary so we can either run, modify or cancel it. One thing to take note of is the Where: line. All files that are restored will have their path prefixed with /tmp/bacula-restores. You could choose another directory or set it to just / if you want Bacula to overwrite the current files in-place. For now accept the current settings by entering yes:

Job queued. JobId=3

Wait a moment and hit Enter to see if Bacula has any news for you. It should:

You have messages.

Let’s look at those:

* mes

You know the job report by now. Look for the following line that shows that everything went right:

Termination: Restore OK

The restore job completed successfully. There are some more useful commands that you can use when you select the files for the restore. I just want to mention two of them: unmark and lsmark. What the former does should be pretty obvious: It deselcts files that were marked for restore before. This allows you to e.g. add * and then unmark a few files which can be a much less painful way if you have more files that are to be restored than files that shouldn’t! The other one shows marked files in and below the current directory. That means if you want to see the full list of marked files, change to / before you use lsmark!

File examination

Let’s quit the bconsole now and take a look at the files that we just recovered from the backup:

* exit
# ls -1 /tmp/bacula-restores/etc/

csh.cshrc
csh.login
csh.logout

Looks like something was restored. Since the original files have not actually been modified since we’ve backed them up, comparing the original and the restored ones should assure us of the files being intact:

# diff -q /usr/local/etc/sudoers /tmp/bacula-restores/usr/local/etc/sudoers

No output means that the files match exactly. Good! But where did those files get restored from? Remember what we did when we configured our backup device. Let’s take a look at the directory that we specified there:

# ls -lh /var/backup/
total 1952
-rw-r—– 1 bacula bacula 1.9M Sep 24 22:08 file3a

This is the volume that we specified in the configuration and that was actually created when we had Bacula label it.

For learning purposes our very simple setup (just one volume) worked great. But before we move on, it’s time to take care of creating a storage system that’s a little bit more advanced: We need a pool! But how do those work?

Volumes, pools and labels

Speaking of labels… In the previous part we had to create one before the job that we queued could actually start. To be able to come up with a sensible backup solution for your use case you will have to understand how Bacula stores backup data. It uses so-called volumes. Think of a volume as some kind of storage medium. This could either be tape or disk-backed storage (i.e. a file). Backup data can be written to a volume until the maximum capacity is reached. Additional data will have to be written to another volume.

We’re not really talking about using tapes here (which comes with its own set of problems from what I ‘ve read in Bacula’s manual). Still it makes sense to remember that tapes are the reason for some design choices of Bacula. Volumes are such a case. While supporting multiple files may not seem like a huge benefit (for one host that is), it’s easy to see that supporting more than one tape does. Once it’s full, write to the next. But to be able to distinguish them, Bacula needs some means of telling them apart. This is where the label comes in. A label basically means that some medium is marked as a volume that Bacula may use combined with a unique name so multiple volumes won’t get confused. So each volume needs a label before Bacula will use it to put data on.

If backup jobs were tied to a volume this could work for some cases but would probably lead to problems sooner or later. Imagine the case when a volume is probably only half full but nevertheless the next backup won’t fit on it. That backup would have to be written to the next volume, wasting the free space on the former. Issues and inflexibilities like that are solved by introducing pools. A pool is basically a list of volumes (plus some options). If your job targets a pool, it no longer matters which volume to put it on – Bacula can take care of that for you in a dynamic way. Pools also allow enforcing some restrictions (like maximum size, maximum time to use) on volumes depending on what your needs are and what you are trying to do.

Since this post is already long enough, it’s time to end this part. As always, let’s save our progress by shutting down the VM and taking the next snapshot (when the status has reached poweroff:

# shutdown -p now
% vagrant status
% vagrant snapshot save tut_4_end

Intermission

After this part of the series we finally know how to restore files from a backup. We also have a better understanding of what jobs, volumes, labels and pools are.

In the next post we’ll create and test a new pool, do some configuration cleanup and reset the catalog. This should conclude the single node part of the tutorial.

Have any comments for me? Or did you perhaps find a mistake? Just leave me a comment.

Bacula on FreeBSD (pt. 3): Customizing configuration

This is the third part of my Bacula tutorial. The first part covered some basics as well as installing Bacula and starting the three daemons that it consists of. Part two dealt with modifying the default configuration files in a way that allowed all components of Bacula to interact with each other locally. And finally a test backup was done (without knowing details like what exactly would even be backed up!) just to ensure that communication between the daemons really works.

In this post we’ll clean up and split the configuration files to make them more comprehensible. We’ll create our own fileset, add a new device and assign that to a new storage resource. We’ll also customize a backup job on the bconsole to do a new backup using our self-created resources.

The third part continues where the second one left off. During this tutorial series we use VirtualBox VMs managed by Vagrant to make things as simple as possible. If you don’t know how to use them, have a look here and here for a detailed introduction into using Vagrant and on how to prepare a VM template with FreeBSD that is used in the Bacula tutorial.

Configuration cleanup

Let’s jump right into it! So restore the snapshot from the previous post, SSH into the machine, become root and change into Bacula’s configuration directory:

% cd ~/vagrant/backuphost
% vagrant snapshot restore tut_2_end
% vagrant ssh
% sudo su -
# cd /usr/local/etc/bacula/

Ok, here we are. Before we can do anything actually somewhat useful, we need to sort out things and get an idea of how the configuration works (what all those resources do, etc). Bacula’s default configuration is rather huge which doesn’t exactly make this an easy task. But there’s an easy solution to it: Let’s start by deleting each and every line that’s commented out in all of the configuration files!

Yes, some of these lines provide useful information but that’s not really lost:

# ls -1
bacula-barcodes
bacula-barcodes.sample
bacula-dir.conf
bacula-dir.conf.sample
bacula-fd.conf
bacula-fd.conf.sample
bacula-sd.conf
bacula-sd.conf.sample
bconsole.conf
bconsole.conf.sample

As you can see, there are files like bacula-dir.conf.sample, bacula-fd.conf.sample, and so on. These are the unaltered default configuration files and you can always look at those as a reference. So before we move on: Edit the four config files that we’ve worked with so far (bacula-dir.conf, bacula-fd.conf, bacula-sd.conf and bconsole.conf) and get rid of all the commented out lines.

Removing these lines (and a few empty ones that were between them and didn’t have any purpose on their own) shrinks the configuration down from over 600 lines to less than 300. While that’s much better, it’s still a lot. Let’s throw more stuff away, shall we? There’s a “restricted director” that’s used for monitoring. We’re not going to use that here so all those resources with the name of fbsd-template.local-mon can be deleted from all three daemon’s config files. Do that now.

At this point restart all three daemons (and check if they are running afterwards) to make sure the configuration is still valid:

# service bacula-dir restart
# service bacula-fd restart
# service bacula-sd restart

The files bconsole.conf and bacula-fd.conf are trivial now with less than 20 lines each. Still bacula-sd.conf with about 80 lines is barely acceptable and especially bacula-dir.conf with over 170 remains too big for my taste!

Configuration splitting

Of course we cannot simply go rampage and delete more and more resources from the director configuration. So how to make it smaller (while maintaining the functionality)? Bacula supports breaking up larger configuration files. To do so we put some resources into their own file and include that in the main config file! Bacula doesn’t care in which files you stuff its resources and it also doesn’t really care about the order. But we do that to make things easier to maintain and so it makes sense to think about what resources should be grouped together.

To keep things organized, it’s best to create a separate directory for includes (or even multiple directories if you have complex configuration and need to separate things further). Choose a name that fits your taste:

# mkdir includes

Here’s my suggestion (you might want to do this completely differently – just do what makes sense for you) for splitting bacula-dir.conf:

  • Move all JobDefs and Job resources into includes/dir_job.conf
  • Move all FileSet resources into includes/dir_fileset.conf
  • Move all Schedule resources into includes/dir_schedule.conf
  • Move all Storage resources into includes/dir_storage.conf
  • Move all Pool resources into includes/dir_pool.conf

Now let’s include those configuration files that we split off! We could do so by including them one by one by adding the following lines at the end of bacula-dir.conf like this:

@/usr/local/etc/bacula/includes/dir_jobs.conf
@/usr/local/etc/bacula/includes/dir_filesets.conf
@/usr/local/etc/bacula/includes/dir_schedule.conf
@/usr/local/etc/bacula/includes/dir_storage.conf
@/usr/local/etc/bacula/includes/dir_pool.conf

That would work (just make sure you use absolute paths to save you some potential headaches). But don’t put it in there (or remove it again if you already have)! There’s a better way of doing it; just add the one following line instead:

@|"sh -c 'for f in /usr/local/etc/bacula/includes/dir_*.conf ; do echo @${f} ; done'"

This will use the shell to automatically include all files in /usr/local/etc/bacula/includes/ that start with dir_ and end in .conf. Much better, eh? And it has the advantage of automatically pulling in includes/dir_messages.conf or something like that should you ever decide to split e.g. the Messages resources from the main dir config file as well.

Now we have a bunch of files with nicely separated resource types and the biggest ones are around 40 lines long. Much more comprehensible if you ask me. Let’s quickly split up the sd config as well by just putting all the Device resources into include/sd_device.conf and including it adding the following line into bacula-sd.conf:

@|"sh -c 'for f in /usr/local/etc/bacula/includes/sd_*.conf ; do echo @${f} ; done'"

Restart dir and sd again to see if the configuration is still ok:

# service bacula-dir restart
# service bacula-sd restart

Hostname

We’ve given our base box a hostname of “fbsd-template.local”. That’s fine for a template – but this machine is not a template anymore. We’ll spin up more VMs from the same base box (template) later so it makes sense to do some proper naming. We could just change the hostname and Bacula would not care at all here since a resource’s name is just a name. But for human operators things can get pretty confusing pretty fast – and that’s what we wanted to avoid, right? So let’s edit the four config files now and change the Name of the various resources from e.g. fbsd-template.local-dir to backuphost.local-dir. Once you’ve replaced it nine times in the actual config files, you’re done here.

But we must not forget to also change it twice in includes/dir_job.conf or the director won’s start since the jobs refer to the old name! And while we’re at it, let’s also change the first Job resource’s name in the same file from BackupClient1 to Backuphost.local. Save the file and exit the editor.

Now let’s change the hostname that FreeBSD will assign itself during startup. To do so we need to edit rc.conf:

# vi /etc/rc.conf

Change the corresponding line so that it holds the new hostname:

hostname="backuphost.local"

Now is an excellent time to take another snapshot (so you can restore that if something goes wrong and don’t have to redo all the config splitting, etc.) and since we’ve changed the hostname we need to reboot anyways. So let’s shutdown the VM, snapshot and start the VM again so that it comes back up with the new hostname:

# shutdown -p now
% vagrant status
% vagrant snapshot save tut_3_mid
% vagrant up

Let’s SSH back into it, switch to root and see if the hostname changed:

% vagrant ssh
% sudo su -
# hostname

backuphost.local

That looks good. Also all Bacula services should be running (feel free to double-check) so we can move on.

What to back up: Filesets

Let’s talk about FileSets next. We’ve put definitions of this kind of resources into their own configuration file that’s then included into the dir configuration. Now we want to add a new fileset. We do so by editing:

# cd /usr/local/etc/bacula
# vi includes/dir_fileset.conf

Just add the following lines to it:

FileSet {
Name = "etc"
Include {
Options {
signature = MD5
}
File = /etc
File = /usr/local/etc
}
Exclude {
File = /etc/casper
}
}

What does this do? It defines another fileset named “etc” which includes the directory /etc. By default Bacula will recursively include files and subdirectories when it is told to back up a directory. So this fileset will effectively backup almost all of the system’s configuration files (which reside in /etc and /usr/local/etc).

Let’s assume we don’t use Capsicum (FreeBSD’s sandboxing framework in case you’ve never heard of it) and don’t want to back up configuration for the Casper daemon (which is a support daemon for Capsicum). The above example excludes /etc/casper from being backed up.

And then there’s the Signature directive. It tells Bacula to create a signature for each file it backs up using the MD5 algorithm. While it is not necessary it is strongly recommended to let Bacula calculate signatures (checksums). Bacula uses these to find out if a file has been altered since it was last backed up even if e.g. the file size didn’t change. Even though it takes a little more space if you use signatures, doing so totally makes sense.

Save the file and exit. You don’t have to restart the director, yet, because we’re going to make one more change.

Where to back up to: Storage

Let’s say we want to store our backups in /var/backup. To do this, we first create the directory and make sure Bacula can access it:

# mkdir /var/backup
# chown bacula:bacula /var/backup

Now we have to inform the sd of the new “device”. Edit the configuration:

# vi includes/sd_device.conf

and add the following lines:

Device {
Name = Var-File
Media Type = File3
Archive Device = /var/backup
LabelMedia = yes;
Random Access = Yes;
AutomaticMount = yes;
RemovableMedia = no;
AlwaysOpen = no;
Maximum Concurrent Jobs = 5
}

Here we create a new “device” for Bacula to use – this “device” is in our case just a directory (/var/backup) on the disk. It is important that you use unique media types; since there’s already File1 and File2 in the default configuration, we go with File3 here. In fact you could choose any unique name that makes some sense to you. Ignore the other settings (or look them up in the Bacula documentation if you want to know more). Save the file and exit. However you won’t be able to use it, yet, as the dir does not know anything about it. So we need to edit its configuration as well:

vi includes/dir_storage.conf

and add the following lines:

Storage {
Name = File3
Address = localhost
SDPort = 9103
Password = "sdPASSWORD"
Device = Var-File
Media Type = File3
Maximum Concurrent Jobs = 10
}

This creates a new storage that can hold backups. It is defined for the director who will send it to the fd if it initiates a backup. Save and exit. Now we need to restart dir and sd to load the new configuration:

# service bacula-dir restart
# service bacula-sd restart

Backup time!

We’ve made a lot of changes to the configuration so let’s attempt to do another backup:

# bconsole
* run

Automatically selected Catalog: MyCatalog
Using Catalog “MyCatalog”
A job name must be specified.
The defined Job resources are:
1: Backuphost.local
2: BackupCatalog
3: RestoreFiles
Select Job resource (1-3):

Select the first one:

Run Backup job
JobName: Backuphost.local
Level: Incremental
Client: backuphost.local-fd
FileSet: Full Set
Pool: File (From Job resource)
Storage: File1 (From Job resource)
When: 2016-09-23 18:36:24
Priority: 10
OK to run? (yes/mod/no):

Choose mod this time:

Parameters to modify:
1: Level
2: Storage
3: Job
4: FileSet
5: Client
6: When
7: Priority
8: Pool
9: Plugin Options
Select parameter to modify (1-9):

Select 4:

The defined FileSet resources are:
1: Full Set
2: Catalog
3: etc
Select FileSet resource (1-3):

Of course we want to use 3, then mod and then 2:

The defined Storage resources are:
1: File1
2: File2
3: File3
Select Storage resource (1-3):

Choose 3, then mod and then 8:

The defined Pool resources are:
1: Default
2: File
3: Scratch
Select Pool resource (1-3):

Go with the first one:

Run Backup job
JobName: Backuphost.local
Level: Incremental
Client: backuphost.local-fd
FileSet: etc
Pool: Default (From User input)
Storage: File3 (From user selection)
When: 2016-09-23 18:38:24
Priority: 10
OK to run? (yes/mod/no):

Now type in yes:

Job queued. JobId=2
You have messages.

Let’s see how that went:

* mes
[…]
23-Sep 18:41 backuphost.local-dir JobId 2: Using Device “Var-File” to write.
23-Sep 18:41 backuphost.local-sd JobId 2: Job Backuphost.local.2016-09-23_18.41.29_03 is waiting. Cannot find any appendable volumes.
Please use the “label” command to create a new Volume for:
Storage: “Var-File” (/var/backup)
Pool: Default
Media type: File3

Oh my! What’s happening now again? We need to create a label first before Bacula can use the new storage. Let’s just do that now and talk about the detail later (there’s a bit more to storage actually):

* label
The defined Storage resources are:
1: File1
2: File2
3: File3
Select Storage resource (1-3):

Select target 3 of course. Bacula will then ask you to give that volume a name:

Enter new Volume name:

Let’s just call it file3a:

Defined Pools:
1: Default
2: File
3: Scratch
Select the Pool (1-3):

Select 1:

Connecting to Storage daemon File3 at localhost:9103 …
Sending label command for Volume “file3a” Slot 0 …
3000 OK label. VolBytes=212 VolABytes=0 VolType=1 Volume=”file3a” Device=”Var-File” (/var/backup)
Catalog record for Volume “file3a”, Slot 0 successfully created.
Requesting to mount Var-File …
3001 OK mount requested. Device=”Var-File” (/var/backup)

Seems like everything is ok. Wait a few seconds and see if the job that has been waiting in line has been run automatically:

* mes
[ … quite some output … ]

Again look especially for the most important line. If you can find this one:

Termination: Backup OK

everything has worked out.

Let’s save our progress by shutting down the VM and taking the next snapshot:

* exit
# shutdown -p now
% vagrant status
% vagrant snapshot save tut_3_end

Intermission

Alright! We just did our second backup with Bacula and this time we actually even know what we did there. While it’s not perfect to modify the backup job like this we’re certainly making progress (and it makes sense to have done it like that at least once). We’ll take care of the job defaults, discuss labels, volumes, etc. next time and also do a restore. But for now take a break. You’ve earned it.

Got any comments or figured that there’s something wrong with what I wrote? Just drop me a line in the comments.

Bacula on FreeBSD (pt. 2): Bconsole – ruling the Director

This is the second part of my hands-on Bacula tutorial. The first part introduced Bacula for people without any previous knowledge, discussed some basics and covered getting all three daemons up and running. This post will explain how to use bconsole to connect to the director, ensure that the director can connect to the other two daemons and show how to do a simple backup. Plus we’ll create an invalid configuration on purpose and debug the issue.

The second part continues where the first one left off. During this tutorial series we use VirtualBox VMs managed by Vagrant to make things as simple as possible. If you don’t know how to use them, have a look here and here for a detailed introduction into using Vagrant and on how to prepare a VM template with FreeBSD that is used throughout all parts of this Bacula tutorial.

Excursion: Vagrant snapshots

We have to start the VM and SSH into it so we can continue setting up Bacula. While it’s not actually required if you didn’t do anything else with the VM in the meantime, it doesn’t hurt to just restore the snapshot that we took at the end of the previous part. But what are snapshots anyway?

A snapshot is a recording of the state of a VM at a specific point in time. You can create a snapshot of any VM; that VM may be running or powered off when the snapshot is created. Snapshots basically hold filesystem contents and (in case of a running VM) system memory, etc.

Snapshots allow you to return to the state that a VM had at the moment that the snapshot was created. If you deleted some files after making the snapshot, returning to the previous state means that the files are back again. At the same time any files that were created after the snapshot are gone and e.g. configuration files will lose any changes made to them. If you created a snapshot of a running VM, returning to the previous state also means that any processes that you stopped after sharpshooting will be running again and so on.

Snapshots work by recording the differences between the state that they captured and the current one. That way they do not double the space that the VM needs (this is what a full clone does, BTW.). The more the states differ however, the more space the snapshot will require. You can keep multiple snapshots for a VM and change back and forth between them. Just keep in mind that they can eat up free space quite fast and are not tied to the disk size quota of the VM itself!

To list all available snapshots (for the particular VM) use the following command:

% vagrant snapshot list

If you want to delete an old snapshot use:

% vagrant snapshot delete SNAPSHOTNAME

Now restore the snapshot from the previous post, SSH into the machine and switch to root:

% cd ~/vagrant/backuphost
% vagrant snapshot restore tut_1_end
% vagrant ssh
% sudo su -

Connecting to the director

We left off discussing configuration – and once it’s configured properly, Bacula is meant to run on its own with little to no manual intervention. However we’re nowhere near a proper configured Bacula, yet! So the first thing to do is to ensure that manual intervention is possible at all.

We control the dir (which in turn controls both the sd and the fd) using the Bacula console or bconsole. There are other programs available but bconsole is a text mode utility that comes with Bacula and will be our default way of interacting with the daemons. Let’s try out what happens if we invoke the console:

# bconsole
Director authorization problem.
Most likely the passwords do not agree.

[…]

We can’t connect. Bacula uses passwords to prove that one component of it is allowed to connect to another. These passwords are randomly generated when Bacula is installed and right now the director has a password set that’s different from the one in bconsole’s configuration. So let’s edit the director’s config file now:

# cd /usr/local/etc/bacula/
# vi bacula-dir.conf

Look for the Director resource with the name fbsd-template.local-dir. Change the Password directive to Password = "dirPASSWORD now. Save the file and exit the editor. And yes, the missing double quote is intentional; you’ll probably ruin your config multiple times as you follow along this series (or try to figure out things yourself) so it makes sense to show you right now how to debug configuration issues!

Edit the bconsole config file next:

# vi bconsole.conf

It should only have the Director resource with the name fbsd-template.local-dir. Change the Password directive there to the same password (however with the ending double quote!). Save and exit.

The director daemon needs to be restarted to pick up the configuration changes. Issue to following command to do so:

# service bacula-dir restart
# service bacula-dir status

bacula_dir is not running.

Our config is broken and so the director won’t start. BTW: Form a habit of checking that the daemon actually runs after making config changes and restarting it! I won’t explicitly mention it from here on. Just get used to always doing it yourself.

Debugging config errors

Let’s assume for a moment that we don’t know of the invalid config file. How do we learn about it? The director won’t start so let’s take a look at the log, shall we?

# cat /var/log/bacula.log
[…Just some old, unrelated messages…]

Hm! There’s nothing new in there?! Only message indicating the failed attempt of bconsole to connect to the dir and even older stuff? Yes, that’s true. Unfortunately the director seems to open the log file and to post messages into it only after it processed the configuration file. And if it runs into a parser error while doing so, it will terminate. So what do we do now?

In such cases it’s a good idea to start the director daemon manually in verbose and debug mode to test the configuration:

# bacula-dir -v -d 1 -c /usr/local/etc/bacula/bacula-dir.conf
bacula-dir: ERROR TERMINATION at parse_conf.c:465
Config error: Could not find config Resource DefaultJob referenced on line 50 : JobDefs = “DefaultJob”
: line 50, col 24 of file /usr/local/etc/bacula/bacula-dir.conf
JobDefs = “DefaultJob”

There we have it: It’s a config error in line 50. Hit CTRL+C to quit the process and take a look at the config file. But what’s that? Line 50 is a perfectly fine directive! Why does the director give that error? Read again and read carefully. The problem is NOT line 50. The problem is that this line references a resource named DefaultJob that is not present.

Search for it in the config file and you’ll find it (beginning on line 29). And worse: it’s perfectly fine as well! Strange, isn’t it? Well, yes and no. From a human perspective: Certainly. But look at the lines above and you’ll see line 25 with the missing double quote. That means that Bacula treats all following characters and lines until it finally finds the next double quote as part of the Password directive… That includes the resource type JobDefs and its name DefaultJob!

That’s why Bacula rightfully claims that a resource with that name wasn’t defined anywhere in the configuration file. And that’s just one “nice” example of how complicated it sometimes is to repair things once they start going sideways!

Now let’s fix the missing double quote in bacula-dir.conf real quick, restart the daemon and continue on:

# vi bacula-dir.conf
# service bacula-dir start

Bconsole

So can we finally connect now?

# bconsole
Connecting to Director localhost:9101
1000 OK: 102 fbsd-template.local-dir Version: 7.4.2 (06 June 2016)

Excellent! We’re eventually connected to the director. The asterisk (*) is the prompt symbol. We can now use the console to issue commands to the director. Let’s try one:

* status dir
fbsd-template.local-dir Version: 7.4.2 (06 June 2016) amd64-portbld-freebsd11.0 freebsd 11.0-RC3
Daemon started 20-Sep-16 20:26, conf reloaded 20-Sep-2016 20:26:01

[…]

OK, there’s some output about the director itself, about jobs and so on. The director seems to be alright. Now for the file daemon:

* status client
Automatically selected Client: fbsd-template.local-fd
Connecting to Client fbsd-template.local-fd at localhost:9102
Failed to connect to Client fbsd-template.local-fd.
====
You have messages.

So connecting to the file daemon failed! And we have messages. Let’s look at the latter first:

* mes
[…]
20-Sep 20:33 fbsd-template.local-dir JobId 0: Fatal error: Unable to authenticate with File daemon at “localhost:9102”. Possible causes:
Passwords or names not the same or
Maximum Concurrent Jobs exceeded on the FD or
FD networking messed up (restart daemon).

The command mes is just short for messages which would also work and do the same. The error here is that the connection was refused – we need to check passwords again. But before doing so, we need to quit bconsole first:

* exit

File daemon configuration

Time to edit the fd configuration file:

# vi bacula-fd.conf

Find the Director resource with the name fbsd-template.local-dir and change the password to "fdPASSWORD". Save and exit. Then edit the dir configuration again:

# vi bacula-dir.conf

Look for the Client resource with the name of fbsd-template.local-fd and change the password to "fdPASSWORD" as well. Save the file and exit the editor. Now restart both the fd and the dir:

# service bacula-fd restart
# service bacula-dir restart

Let’s see if that fixed the problem:

# bconsole
* status client

Automatically selected Client: fbsd-template.local-fd
Connecting to Client fbsd-template.local-fd at localhost:9102

fbsd-template.local-fd Version: 7.4.2 (06 June 2016) amd64-portbld-freebsd11.0 freebsd 11.0-RC3
Daemon started 20-Sep-16 20:51. Jobs: run=0 running=0.

All looking good! Now only the sd remains. Let’s give that a password that fits the scheme, too.

Storage daemon configuration

Now for the sd config:

* exit
# vi bacula-sd.conf

Go to the Director resource with the name fbsd-template.local-dir and set the password to "sdPASSWORD" this time. Save changes and exit. Next is the dir config once again:

# vi bacula-dir.conf

Find the Storage resources (named File1 and File2) and set their password to "sdPASSWORD" (and ignore that the configuration file says that you should not use localhost for sd while it configures it to use localhost by default…). Save and exit.

Now restart the sd and the dir and try to connect to the sd using bconsole:

# service bacula-sd restart
# service bacula-dir restart
# bconsole
* status sd

Automatically selected Storage: File1
Connecting to Storage daemon File1 at localhost:9103

fbsd-template.local-sd Version: 7.4.2 (06 June 2016) amd64-portbld-freebsd11.0 freebsd 11.0-RC3
Daemon started 20-Sep-16 21:09. Jobs: run=0, running=0.

Great, now bconsole can connect to the dir and the dir to both the fd and the sd!

Doing a backup

Before finally doing a first test backup, we’re going to edit the director config once more to tweak messaging a bit:

* exit
# vi bacula-dir.conf

Skip down to the Messages resource named Standard and find the following four lines:

mailcommand = "/usr/local/sbin/bsmtp -h localhost -f \"\(Bacula\) \\" -s
\"Bacula: %t %e of %c %l\" %r"
operatorcommand = "/usr/local/sbin/bsmtp -h localhost -f \"\(Bacula\) \\"
-s \"Bacula: Intervention needed for %j\" %r"
mail = root@localhost = all, !skipped
operator = root@localhost = mount

By default, Bacula sends mails if errors occur. We haven’t configured mailing, so this would only result in more errors… Since we’re not building a production environment here, let’s just get rid of this and make life a bit easier while exploring the essential functionality of Bacula. Comment out all four. Then save the file, exit and restart the director once again:

# service bacula-dir restart

Now issue the following commands:

# bconsole
* run

Automatically selected Catalog: MyCatalog
Using Catalog “MyCatalog”
A job name must be specified.
The defined Job resources are:
1: BackupClient1
2: BackupCatalog
3: RestoreFiles
Select Job resource (1-3):

Pick number 1.

Run Backup job
JobName: BackupClient1
Level: Incremental
Client: fbsd-template.local-fd
FileSet: Full Set
Pool: File (From Job resource)
Storage: File1 (From Job resource)
When: 2016-09-20 21:24:13
Priority: 10
OK to run? (yes/mod/no):

Answer “yes”.

Job queued. JobId=1
You have messages.

Ask the director to display the messages:

* mes

You should get a lot of output. If you only got a few lines, you probably were too fast and caught Bacula while it was still working. Give it a second and type mes again. The important line to look for is this one near the end:

Termination: Backup OK

Take a look at all the lines and get a rough idea what has happened here. Do so especially if at this point something bad happened (i.e. the backup failed). In this case check that you did everything as I wrote it here and if that doesn’t help, you’re unfortunately in for some research… But I’ll assume that it worked well and you just did your first Bacula backup. Good work! This means that you’ve also successfully ensured that all daemons can interact with each other.

That’s it for this part of the tutorial. Power down the VM and save another snapshot (when the status has reached poweroff):

* exit
# shutdown -p now
% vagrant status
% vagrant snapshot save tut_2_end

Intermission

We’re still in the early stages of setting up Bacula but at least this time we were able to actually do something by backing up some data. You probably have no idea what you’ve just backed up or where to – but that’s ok for now. We’ll discuss these things in the next part. And in addition to making all daemons able to interact and doing a backup, you’ve done some debugging work. So I think having a break here and talking about backups in more detail later (I also want to cover configuration some more before I do so) makes sense.

I hope that this has been a useful read. Feel free to leave me a comment if you like it (or if you don’t) and of course tell me if I made some mistakes that should be corrected!

Bacula on FreeBSD (pt. 1): Introduction – Bacula backup basics

Looking for a practical tutorial to get into Bacula hands-on? You’ve come to the right place. So this post is going to cover the three “B”: Bacula Backup Basics, and it will do so on a fourth “B” – (Free)BSD! Read “basics” auds “does not require previous knowledge”. However it’s Bacula basics. Even though the project’s manual officially discourages (!) using Bacula for people unfamiliar with “sophisticated backup tools”, don’t be afraid of it. The reason for the statement in the manual is that Bacula requires quite a bit of configuration work before it can do anything useful. Due to its modularity it’s also a bit harder to understand compared to simple backup tools.

All of that is true. That’s why we won’t be doing backups in this first part. Instead I’ll discuss some basics here that you definitely have to know. We’ll also install Bacula and get all of its parts running. That’s in fact enough for a single post.

While this post does not expect you to know anything about Bacula, you should have a general understanding of backup fundamentals. Simple test: If you know the terms „incremental backup“ and „differential backup“ and you know what both mean, chances are good that you’ll be fine. If you don’t know them, please do a bit of research first and come back later.

What is Bacula?

Bacula is an Open Source backup tool with a lot of advanced features (which I don’t claim to know much about). Or more precisely: Bacula is a set of tools to perform backups. As I touched upon above, Bacula is modular and it is crucial to know its parts:

  • bacula-dir
  • bacula-sd
  • bacula-fd
  • bconsole
  • catalog

First we have bacula-dir, the Bacula director daemon. Just like the name suggests, this is the central instance that supervises and coordinates backup work.

Then there’s bacula-sd, the Bacula storage daemon. The name also kind of implies what it does: It is responsible for storing the backup data that it receives or for reading and sending it in case of a restore.

The third daemon, bacula-fd, is the Bacula file daemon. I’d prefer to call it “client daemon” as that would in my opinion fit it better, but that’s not what it is called. What does it do? It reads files from a backup client and sends them to the storage daemon (sd). Or, in case of a restore, the file daemon (fd) receives data from the sd and writes it back to the disk.

Not a daemon but rather a small administration utility is bconsole. It provides a way of connecting to the director daemon (dir) and interact with it. You need it for maintenance, to debug problems or to test new configuration.

The catalog is where Bacula stores various information about backups (NOT the backup data itself!). We’re just starting with Bacula, so let’s keep things simple. All you have to know for now is that Bacula needs a catalog and that it is kept in a database (MySQL, PostgreSQL and SQLite are supported).

Ok, so much for Bacula’s structure. Next station: The practical part, Installing and getting it to work! But one more thing first: Whenever something about Bacula doesn’t seem to make sense to you, here’s a clue for you. Bacula was designed with tape backups in mind. Recalling this fact often helped me understanding why something in Bacula is the way that it is and not built differently!

Installation

My previous post showed how to prepare a Vagrant base box for playing with Bacula. I’ll assume that you are either using that or have a FreeBSD system available that’s configured likewise. You can ignore the Vagrant stuff in that case. I do recommend however that you use Vagrant. For now we’ll stick with a simple setup of one node, but once we add more nodes to our setup in later parts of this series, Vagrant comes in really, really handy. If you don’t know the tool, I’ve written another post that introduces Vagrant + VirtualBox for beginners.

Let’s create a directory for the VM “backuphost” first and populate it with a default Vagrantfile:

% cd ~/vagrant
% mkdir backuphost
% cd backuphost
% vagrant init

Edit the Vagrantfile, delete its content and put the following into it:

Vagrant.configure("2") do |config|
config.vm.box = "fbsd-template"
config.vm.network "private_network", type: "dhcp"
config.vm.synced_folder ".", "/vagrant", disabled: true
end

The network setting (creating a private network) is not needed right now but we’re setting it up already so that it can simply be used when we need it later. Now fire up the VM and SSH into it:

% vagrant up
% vagrant ssh

We’ve already prepared things during the base box creation. Now become root and install bacula-server:

% sudo su -
# cd /usr/ports/sysutils/bacula-server/
# make install clean

FileDaemon and StorageDaemon

Now that Bacula is installed, let’s see if we can get the various daemons to run. If you know what they are called, that’s usually not too hard. We’ll start with launching the fd:

# sysrc bacula_fd_enable="YES"
# service bacula-fd start

Starting bacula_fd.

Fair enough. But has it started successfully? Let’s check:

# service bacula-fd status
bacula_fd is running as pid 14003.

It has, everything’s looking fine here. Ignore the PID, BTW, it will most likely be different for you. Time to enable, start and check the sd next:

# sysrc bacula_sd_enable="YES"
# service bacula-sd start

Starting bacula_sd.
# service bacula-sd status
bacula_sd is running as pid 14056.

Again: Nothing unexpected happening here. The default configuration that comes with Bacula on FreeBSD is sufficient for both daemons to be happy with and start up. Now for the remaining daemon, the director… So far things were downright boring. But now they get a bit more interesting (you decide if that’s a good thing. Most administrators hate it when software behaves in interesting ways).

Bringing up the director

Let’s start the director daemon after enabling it for system start and see if it’s running:

# sysrc bacula_dir_enable="YES"
# service bacula-dir start

Starting bacula_dir.
# service bacula-dir status
bacula_dir is not running.

It isn’t! What could be wrong here? What do you do if you have no clue? Take a look at the system log of course:

# tail /var/log/messages | grep bacula-dir
Sep 20 19:00:00 fbsd-template bacula-dir: 20-Sep 19:00 Message delivery ERROR: fopen /var/log/bacula.log failed: ERR=Permission denied

Aha! The director was not allowed to open its log file. Probably because there is no log file! So let’s create it and give it to the bacula user so that it has the permissions needed. Then we’ll try to start the director again:

# touch /var/log/bacula.log
# chown bacula:bacula /var/log/bacula.log
# service bacula-dir start

Starting bacula_dir.
# service bacula-dir status
bacula_dir is not running.

Too bad! But we should have a log now. Let’s take a look at that:

# tail /var/log/bacula.log
20-Sep 19:11 bacula-dir JobId 0: Fatal error: Could not open Catalog “MyCatalog”, database “bacula”.
20-Sep 19:11 bacula-dir JobId 0: Fatal error: sqlite.c:199 Database /var/db/bacula/bacula.db does not exist, please create it.

Right, we forgot the catalog! I told you to configure it for SQLite when we built the base box so that’s what I’m going to assume here. If you chose MySQL or PostgreSQL, refer to the manual or search the net (PostrgeSQL is the default backend on FreeBSD; so if you installed bacula-server from packages, you’re expected to use that).

Bacula provides scripts for each database backend to created the tables needed. Let’s create the catalog and see if the director can start afterwards:

# su -m bacula
% /usr/local/share/bacula/make_sqlite3_tables
% exit
# service bacula-dir start

Starting bacula_dir.
# service bacula-dir status
bacula_dir is running as pid 14069.

All the daemons are up and running and you know a few essential things about Bacula. However we cannot do anything with it just yet. We need to dig into Bacula’s configuration before we can use the bconsole to interact with the director – which can then interact with the other two daemons.

Bacula configuration files

You have to know a few things before making changes to the default configuration makes sense. As these can be considered “basics”, too, I’ll cover them here.

On FreeBSD, Bacula’s configuration files live in /usr/local/etc/bacula/. Bacula’s configuration is divided into several configuration files – one for each daemon and one for the bconsole (as well as another one that we’ll ignore): bacula-dir.conf, bacula-sd.conf, bacula-fd.conf and bconsole.conf.

There are various passwords in the config files and those can be a little confusing at first. This is why I’ll be using simple passwords that hint which daemon they are actually belonging to. As long as you are just testing Bacula, I recommend you do the same. It’s hard enough to work through Bacula’s configuration and the more clear, easy to understand bits you’ve got in there, the better.

All those files have a common structure. They are composed of definitions which assign a value (or values) to a variable. Definitions which belong together are in the same group. Bacula calls these definitions directives and the groups are known as resources. Since there can be multiple resources of the same type, each one must have a unique name so Bacula can tell them apart. When it comes to resource types, think of them as the context which the resource is meant for (e.g. Director, FileDaemon, Messages, etc.)

The structure of a resource always looks like this:

Type {
directive A
directive B
directive C
[...]
}

In configuration files lines starting with a “#” are commented out. If you see a comment sign later in the line that means that anything from there is ignored as a comment while the part before it isn’t.

Take a look at bconsole.conf and bacula-fd.conf (these are the shortest ones) and get a first impression of what Bacula configuration looks like. Think about some of the resources and guess what they maybe do. Don’t look at bacula-sd.conf or bacula-dir.conf if you’re not looking for some more confusion. If things seem crystal clear for you however, of course feel free to go ahead. It’s also OK if you feel a little overwhelmed after reading the files. We’ll be tearing them apart together in a later part.

Saving the progress

IMPORTANT: DON’T destroy your VM if you intend to follow the other parts, too! We’re not starting fresh each time but instead build upon what we’ve done so far.

Maybe you want to play with Bacula on your own in the meantime. That would however complicate things a lot since the configuration would probably no longer match what I take for granted in the next post. But there’s a simple solution to this: Create a snapshot of your VM now! As long as you don’t remove that snapshot, feel free to do inside the VM as you please.

Shutdown the VM properly first and then use Vagrant to create a snapshot; but before you do, make sure that the VM has reached the poweroff state (if it hasn’t, wait a moment!):

# shutdown -p now
% vagrant status
% vagrant snapshot save tut_1_end

Intermission

That’s it for now. I originally wanted to stuff more into this first post and at least have you do your first backup. But that would have meant to either make this post much longer (which would probably scare some people away) or to skip over some details and have you figure them out yourself (which kind of defeats the idea of writing such a “practical introduction” in the first place).

And since Bacula is a complex topic, I also think that this may be enough information for somebody who’s completely new to it (and perhaps to Vagrant as well). If you feel like this could have been longer you can of course just continue with part 2 whenever you want.

Oh, and if you notice any mistakes or anything, please tell me. I’m by no means a Bacula expert and chances are of course that I’m doing things horribly wrong (it has worked for me so far, though!).

Vagrant: Creating a FreeBSD 11 base box (virtualbox) – pt. 2

The previous post discussed what Vagrant is, explained the installation (on FreeBSD), detailed the process of creating a new VM and briefly described how to install FreeBSD 11. The next step is to prepare the fresh installation for Vagrant and eventually create an actual base box from it. If you’re new to VirtualBox or you don’t know what a provider is when it comes to Vagrant, you might want to read the first part. If you however already have a decent idea of virtualization and making sensible choices when installing FreeBSD, just do a fresh install in a VM and continue reading here.

If you haven’t read the first part: These two posts weren’t written without cause. The idea was to dive into Bacula (a program for doing backups). For testing backups properly you need more than one machine obviously. I don’t have enough spare PCs, so using Virtual Machines seemed like a good way of working around that limitation. Since Vagrant makes working with these VMs really easy, it totally makes sense to use it. I’ve used it quite a bit in the past and while I was at it again, I decided to blog about this great tool so that you maybe consider adding it to your tool set as well.

Preparing the VM for Vagrant: Settings

There are a few things on the side of the guest system that need to be in place so that Vagrant can work with it properly.

The first step is optional but considered good practice if you intend to create a public base box: Disable unneeded resources! To do so, go to the settings of your VM template in the VirtualBox GUI.

Selecting VM settings

A good candidate to disable is audio support which is not generally needed in a Vagrant box.

Disabling audio for the VM

Also deactivate USB support. It’s enabled by default but is most likely unneeded in Vagrant boxes (there are other ways to share data between the VM and the host system).

Disabling USB for the VM

Now fire up your fresh FreeBSD 11 system and log in as root. A little tweak that I suggest is to disable the auto delay of the boot loader. This will make the VM boot up 10 seconds faster because it won’t wait for possible user interaction at the boot loader stage. To do so, you can use the following command:

# echo autoboot_delay="-1" >> /boot/loader.conf

Preparing the VM for Vagrant: Packages

If you’re not using a new version that was just released a few days ago, it’s always a good idea to update the base system first:

# freebsd-update fetch install

Reboot if the kernel was updated:

# shutdown -r now

The FreeBSD package manager (pkg) ist not part of the base system (that way it can evolve faster) so let’s bootstrap it first:

# env ASSUME_ALWAYS_YES=yes pkg install pkg

Vagrant needs sudo to work. And not having bash available works but doesn’t make Vagrant happy. It’s best to just install both:

# pkg install -y sudo bash

Now configure sudo. The configuration file is /usr/local/etc/sudoers on FreeBSD – but you may want to forget that again as you shouldn’t touch that file by hand! Rather issue the command

# visudo

which will call a wrapper script that lets you edit the file and check it for validity before actually saving changes. Why’s that? Because on certain system configurations you can easily lock yourself out of the system with a defunct sudo (and you basically lose sudo when you ruin its configuration file)! Even though the script has vi in its name, that’s just the default. You may want to set the EDITOR environment variable to another editor if you prefer emacs/nano/joe/whatever. Not liking vi is NOT an excuse for not using visudo!

When editing the sudoers file, look for the line %wheel ALL=(ALL) NOPASSWD: ALL. It’s commented out by default. Remove the comment sign and save. That’s actually all that needs to be changed. What that change does: It grants all members of the wheel group unrestricted sudo access without having to enter their password.

Depending on what kind of base box you want to create, you may want to install the VirtualBox guest additions. They enable the hypervisor to directly interact with the guest system enabling some features like a shared clipboard, enhanced graphics options, shared folder support (which are not available on *BSD at the present time however), etc. The downside is that they are quite large (due to their dependencies): ca. 350 MB. If you’re aiming for a small base box, leave them out if you don’t need them. Otherwise consider installing them.

In case you built VirtualBox on your host system from ports you might do the same for the guest additions so that versions match. Building them from ports does complicate things a bit however. You need the system source installed, it will use quite some space and take a long time. Usually it’s not worth the hassle. Unless you require some feature that only works when the versions match (I’ve never run into one), just install the guest additions using packages:

# pkg install virtualbox-ose-additions

Installing the vbox guest additions

If you installed them, enable the guest additions for the next boot:

# sysrc vboxguest_enable="YES"
# sysrc vboxservice_enable="YES"

Preparing the VM for Vagrant: Users

The most important requirement for Vagrant is that a specific user has to be present. In FreeBSD you can create it all by issuing one line:

# echo vagrant | pw adduser -n vagrant -m -s /bin/tcsh -h 0 -L default -c "Vagrant user" -G wheel

This will make pw add a user with the name vagrant to the system, create a home directory for them, set the shell (you can also choose another one here like /usr/local/bin/bash for example) and make it use what is piped into it as the password (that’s a zero following the -h, btw.). It will also set the default login class for the user, use the provided user comment and add it to the wheel group (for which we’ve already configured sudo access).

Now that the vagrant user is present, Vagrant could log you into a VM that it has started from the base box. However it will ask for the password. This is just a small annoyance really, but we can do better. So let’s prepare the vagrant user for automatic login (using SSH keys). To do so, we first change to the new user:

# su -l vagrant

Next we create a .ssh dot directory in user vagrant’s home directory:

% mkdir .ssh

Then we download the public key from the Vagrant insecure keypair (it’s insecure because the private key is public, too – Vagrant uses it to log in via SSH and it’s open source software! But that’s not a problem here) into the new directory e.g. with the following line:

% fetch --no-verify-peer https://raw.githubusercontent.com/mitchellh/vagrant/master/keys/vagrant.pub -o .ssh/authorized_keys

This downloads the vagrant.pub file as authorized_keys (the file that SSH checks against) and skips verifying the SSL certificate of the remote web server (this is necessary because FreeBSD does not ship with a collection of trusted root certificates and we haven’t installed one).

The last thing to do is setting the correct permissions for the new directory and file (SSH is quite picky about that – and rightfully so):

% chmod 0700 .ssh
% chmod 0600 .ssh/authorized_keys

When that’s done, we can log out as vagrant, back to the still logged in root:

% exit

With the key in place we have everything that’s required for a simple FreeBSD 11 base box.

Customizing the base box

Now’s the time to customize your base box. You could install additional software, add more users, do specific configuration, etc. I’m going to prepare the template for testing Bacula with multiple nodes. Since I need to build Bacula with non-standard options, I need to use the ports tree. So let’s put that in /usr/ports first:

# portsnap fetch extract

Then we change into the directory for the Bacula client:

# cd /usr/ports/sysutils/bacula-client

We configure the compile options for the Bacula client as well as for all its dependencies next:

# make config-recursive

I usually deactivate NLS (support for non-English translations), IPv6, DOCS and EXAMPLES for all programs. You may choose different options of course.

Since I need the Bacula client on every machine I can simply build and install it now:

# make install clean

Now let’s change into the Bacula server ports directory:

# cd ../bacula-server

Again we need to set the build options for the port and dependencies:

# make config-recursive

Setting compile options for bacula server

Deselect postgresql and select sqlite instead! When you accept the settings, you will be asked to select the settings for postgresql next. This is because the list of ports to configure was compiled before you changed the dependencies by choosing to build with support for another database. Hit CTRL+C to cancel the operation and then run make config-recursive again.

Just like before, leave out unneeded options to keep the whole system a little lighter. Leave sqlite3 as-is if you don’t know too much about it (it’s a port with A LOT of options and can look rather daunting).

Then we fetch the source archives for Bacula server and its dependencies (this is done because I want to configure the VMs to be restricted to the local network later and thus they won’t be able to connect to the internet then):

# make fetch-recursive

Finalizing the VM

If you’re building a base box that you intend to share with others, it makes sense to try to cut down the size as far as possible. Here are a few suggestions:

  • Clean the pkg cache (removing package files no longer needed after installing them:
    # pkg clean -a
  • Remove the distfiles (/usr/ports/distfiles) if you don’t require them to be available later; if you need some of them, delete the others!
  • If you’ve installed the ports tree to build software and don’t need it in the base box, remove it completely
  • In case you installed the system source (e.g. for building the VirtualBox guest additions) or for customizing the FreeBSD system you probably want to delete /usr/src/* and/or /usr/obj/*
  • Used freebsd-update? Clear any files it may have downlodad or kept for rollback:
    rm -rf /var/db/freebsd-update/*
  • Have installed programs from ports and removed the ports tree again? You can also uninstall programs that were installed as build time dependencies only

Just deleting stuff won’t make your VM’s virtual hard drive smaller during packaging, though! If you’re using a traditional filesystem (i.e. UFS) there’s an easy trick to “reclaim” the now unused space: Zero it out! Don’t try to do this if you’re using ZFS – you won’t be able to fill your drive with zeros if the filesystem compresses them away! So for UFS filesystems do the following:

# dd if=/dev/zero of=/ZERO bs=1m

This will create the file ZERO in / which will be filled with all zeros until the filesystem runs out of space. If you delete that file afterwards, the part of the filesystem that is free again is all zeroed out and thus compresses well when packaging the base box:

# rm /ZERO

Zeroing out free space

If you chose to use multiple filesystems, you may want to dd to a ZERO file on each separate filesystem (e.g. /usr/ZERO, /var/ZERO, etc.).

Finally everything is ready, so shutdown the VM now:

# shutdown -p now

Building the base box

Wait for the VM to power off. Now open a terminal emulator on your host system and tell Vagrant to build a base box from the VM:

% vagrant package --base fbsd-template -o fbsd-template.box

The argument after –base is the name of the VM as it appears in VirtualBox. Mind upper and lower case! Choose any name for the output file (the extension is usually .box). This will take a moment but if all goes well you should eventually have the base box file!

% vagrant box list

This lists all Vagrant boxes available on your system. If you’ve just installed Vagrant before it will tell you that it doesn’t have any boxes yet. So let’s import our newly built box:

% vagrant box add --name fbsd-template fbsd-template.box

Give it any name you want the base box to be known as by Vagrant and point the program to the .box file. If you do

% vagrant box list

Base box created and added!

again, you should see that it’s available now. What’s left? Let’s test if it actually works.

Testing the base box

To do so, I suggest creating a directory for your Vagrant boxes:

% mkdir -p ~/vagrant/testvm
% cd ~/vagrant/testvm

In that directory issue the following command:

% vagrant init

This creates a default Vagrantfile. Think of this as the central configuration for a single VM that Vagrant manages. It contains the name of the base box to create it from, allows configuring of the network, synced folders, overriding base box settings (like the amount of RAM for the VM), provisioning, etc.

Edit this file, have a look at the contents (you can also do this at a later time – vagrant init will always create this default Vagrantfile for you if you need it) and delete it. Put the following lines in the Vagrantfile:

Vagrant.configure("2") do |config|
config.vm.box = "fbsd-template"
config.vm.synced_folder ".", "/vagrant", disabled: true
end

Substitute fbsd-template with the name of your base box if you chose a different one. The second configuration directive disables synced folders: They don’t work for *BSD guests at the present time and Vagrant would issue an unnecessary warning if we didn’t disable it. This is all you need for a very basic configuration. Save the file and exit your editor.

Now run:

% vagrant up

This command tells Vagrant to create a new VM from the base box (VM template) selected in the Vagrantfile and fire it up. Vagrant will do so in the background – it runs the VM in so-called headless mode.

VM created from base box has booted up

You don’t need the VirtualBox GUI for Vagrant but right now take a look. You’ll see a new VM that was created by Vagrant and in fact you could also use VirtualBox to control it. But that’s rarely necessary. Vagrant is meant to manage boxes all for you after all!

To enter the VM, you can execute

% vagrant ssh

Now you’re SSH’ed into the VM! It’s as easy as that. Issue an ls or something to see that it’s working. Exit the machine by logging out (type exit or press CTRL-D) and you’re back in your host machine’s VM directory. Try vagrant ssh again and you see that you’re inside the VM again.

SSH’ed into the box using Vagrant

Cleaning up

Shutdown the VM now:

% sudo shutdown -p now

When the VM shuts down the SSH daemon, you’re automatically dropped back to your host system. Let’s quickly do some cleaning up and we’re done for now:

% vagrant destroy

This will purge the VM. If you keep the Vagrantfile, you can just vagrant up afterwards to start over with a clean VM (from the base box) again. This is great for testing things. You can also control the VM from outside using commands like vagrant halt, vagrant suspend, vagrant resume, etc. I suggest you take a look yourself and learn what Vagrant can do for you. It really has some neat and useful features.

Just a

% cd ..
% rm -r testvm

later we’re done with cleaning up after ourselves. The base box works and can be used to quickly spin up VMs based off of it. While this is neat, the real power Vagrant has and the real comfort it provides show when you’re working with several VMs at the same time. Which is what I intend to do in the post after the next one.

What’s next?

The next post will be an introduction to the Bacula backup software, exploring the basic functionality.