Bacula on FreeBSD (pt. 2): Bconsole – ruling the Director

This is the second part of my hands-on Bacula tutorial. The first part introduced Bacula for people without any previous knowledge, discussed some basics and covered getting all three daemons up and running. This post will explain how to use bconsole to connect to the director, ensure that the director can connect to the other two daemons and show how to do a simple backup. Plus we’ll create an invalid configuration on purpose and debug the issue.

The second part continues where the first one left off. During this tutorial series we use VirtualBox VMs managed by Vagrant to make things as simple as possible. If you don’t know how to use them, have a look here and here for a detailed introduction into using Vagrant and on how to prepare a VM template with FreeBSD that is used throughout all parts of this Bacula tutorial.

Excursion: Vagrant snapshots

We have to start the VM and SSH into it so we can continue setting up Bacula. While it’s not actually required if you didn’t do anything else with the VM in the meantime, it doesn’t hurt to just restore the snapshot that we took at the end of the previous part. But what are snapshots anyway?

A snapshot is a recording of the state of a VM at a specific point in time. You can create a snapshot of any VM; that VM may be running or powered off when the snapshot is created. Snapshots basically hold filesystem contents and (in case of a running VM) system memory, etc.

Snapshots allow you to return to the state that a VM had at the moment that the snapshot was created. If you deleted some files after making the snapshot, returning to the previous state means that the files are back again. At the same time any files that were created after the snapshot are gone and e.g. configuration files will lose any changes made to them. If you created a snapshot of a running VM, returning to the previous state also means that any processes that you stopped after sharpshooting will be running again and so on.

Snapshots work by recording the differences between the state that they captured and the current one. That way they do not double the space that the VM needs (this is what a full clone does, BTW.). The more the states differ however, the more space the snapshot will require. You can keep multiple snapshots for a VM and change back and forth between them. Just keep in mind that they can eat up free space quite fast and are not tied to the disk size quota of the VM itself!

To list all available snapshots (for the particular VM) use the following command:

% vagrant snapshot list

If you want to delete an old snapshot use:

% vagrant snapshot delete SNAPSHOTNAME

Now restore the snapshot from the previous post, SSH into the machine and switch to root:

% cd ~/vagrant/backuphost
% vagrant snapshot restore tut_1_end
% vagrant ssh
% sudo su -

Connecting to the director

We left off discussing configuration – and once it’s configured properly, Bacula is meant to run on its own with little to no manual intervention. However we’re nowhere near a proper configured Bacula, yet! So the first thing to do is to ensure that manual intervention is possible at all.

We control the dir (which in turn controls both the sd and the fd) using the Bacula console or bconsole. There are other programs available but bconsole is a text mode utility that comes with Bacula and will be our default way of interacting with the daemons. Let’s try out what happens if we invoke the console:

# bconsole
Director authorization problem.
Most likely the passwords do not agree.

[…]

We can’t connect. Bacula uses passwords to prove that one component of it is allowed to connect to another. These passwords are randomly generated when Bacula is installed and right now the director has a password set that’s different from the one in bconsole’s configuration. So let’s edit the director’s config file now:

# cd /usr/local/etc/bacula/
# vi bacula-dir.conf

Look for the Director resource with the name fbsd-template.local-dir. Change the Password directive to Password = "dirPASSWORD now. Save the file and exit the editor. And yes, the missing double quote is intentional; you’ll probably ruin your config multiple times as you follow along this series (or try to figure out things yourself) so it makes sense to show you right now how to debug configuration issues!

Edit the bconsole config file next:

# vi bconsole.conf

It should only have the Director resource with the name fbsd-template.local-dir. Change the Password directive there to the same password (however with the ending double quote!). Save and exit.

The director daemon needs to be restarted to pick up the configuration changes. Issue to following command to do so:

# service bacula-dir restart
# service bacula-dir status

bacula_dir is not running.

Our config is broken and so the director won’t start. BTW: Form a habit of checking that the daemon actually runs after making config changes and restarting it! I won’t explicitly mention it from here on. Just get used to always doing it yourself.

Debugging config errors

Let’s assume for a moment that we don’t know of the invalid config file. How do we learn about it? The director won’t start so let’s take a look at the log, shall we?

# cat /var/log/bacula.log
[…Just some old, unrelated messages…]

Hm! There’s nothing new in there?! Only message indicating the failed attempt of bconsole to connect to the dir and even older stuff? Yes, that’s true. Unfortunately the director seems to open the log file and to post messages into it only after it processed the configuration file. And if it runs into a parser error while doing so, it will terminate. So what do we do now?

In such cases it’s a good idea to start the director daemon manually in verbose and debug mode to test the configuration:

# bacula-dir -v -d 1 -c /usr/local/etc/bacula/bacula-dir.conf
bacula-dir: ERROR TERMINATION at parse_conf.c:465
Config error: Could not find config Resource DefaultJob referenced on line 50 : JobDefs = “DefaultJob”
: line 50, col 24 of file /usr/local/etc/bacula/bacula-dir.conf
JobDefs = “DefaultJob”

There we have it: It’s a config error in line 50. Hit CTRL+C to quit the process and take a look at the config file. But what’s that? Line 50 is a perfectly fine directive! Why does the director give that error? Read again and read carefully. The problem is NOT line 50. The problem is that this line references a resource named DefaultJob that is not present.

Search for it in the config file and you’ll find it (beginning on line 29). And worse: it’s perfectly fine as well! Strange, isn’t it? Well, yes and no. From a human perspective: Certainly. But look at the lines above and you’ll see line 25 with the missing double quote. That means that Bacula treats all following characters and lines until it finally finds the next double quote as part of the Password directive… That includes the resource type JobDefs and its name DefaultJob!

That’s why Bacula rightfully claims that a resource with that name wasn’t defined anywhere in the configuration file. And that’s just one “nice” example of how complicated it sometimes is to repair things once they start going sideways!

Now let’s fix the missing double quote in bacula-dir.conf real quick, restart the daemon and continue on:

# vi bacula-dir.conf
# service bacula-dir start

Bconsole

So can we finally connect now?

# bconsole
Connecting to Director localhost:9101
1000 OK: 102 fbsd-template.local-dir Version: 7.4.2 (06 June 2016)

Excellent! We’re eventually connected to the director. The asterisk (*) is the prompt symbol. We can now use the console to issue commands to the director. Let’s try one:

* status dir
fbsd-template.local-dir Version: 7.4.2 (06 June 2016) amd64-portbld-freebsd11.0 freebsd 11.0-RC3
Daemon started 20-Sep-16 20:26, conf reloaded 20-Sep-2016 20:26:01

[…]

OK, there’s some output about the director itself, about jobs and so on. The director seems to be alright. Now for the file daemon:

* status client
Automatically selected Client: fbsd-template.local-fd
Connecting to Client fbsd-template.local-fd at localhost:9102
Failed to connect to Client fbsd-template.local-fd.
====
You have messages.

So connecting to the file daemon failed! And we have messages. Let’s look at the latter first:

* mes
[…]
20-Sep 20:33 fbsd-template.local-dir JobId 0: Fatal error: Unable to authenticate with File daemon at “localhost:9102”. Possible causes:
Passwords or names not the same or
Maximum Concurrent Jobs exceeded on the FD or
FD networking messed up (restart daemon).

The command mes is just short for messages which would also work and do the same. The error here is that the connection was refused – we need to check passwords again. But before doing so, we need to quit bconsole first:

* exit

File daemon configuration

Time to edit the fd configuration file:

# vi bacula-fd.conf

Find the Director resource with the name fbsd-template.local-dir and change the password to "fdPASSWORD". Save and exit. Then edit the dir configuration again:

# vi bacula-dir.conf

Look for the Client resource with the name of fbsd-template.local-fd and change the password to "fdPASSWORD" as well. Save the file and exit the editor. Now restart both the fd and the dir:

# service bacula-fd restart
# service bacula-dir restart

Let’s see if that fixed the problem:

# bconsole
* status client

Automatically selected Client: fbsd-template.local-fd
Connecting to Client fbsd-template.local-fd at localhost:9102

fbsd-template.local-fd Version: 7.4.2 (06 June 2016) amd64-portbld-freebsd11.0 freebsd 11.0-RC3
Daemon started 20-Sep-16 20:51. Jobs: run=0 running=0.

All looking good! Now only the sd remains. Let’s give that a password that fits the scheme, too.

Storage daemon configuration

Now for the sd config:

* exit
# vi bacula-sd.conf

Go to the Director resource with the name fbsd-template.local-dir and set the password to "sdPASSWORD" this time. Save changes and exit. Next is the dir config once again:

# vi bacula-dir.conf

Find the Storage resources (named File1 and File2) and set their password to "sdPASSWORD" (and ignore that the configuration file says that you should not use localhost for sd while it configures it to use localhost by default…). Save and exit.

Now restart the sd and the dir and try to connect to the sd using bconsole:

# service bacula-sd restart
# service bacula-dir restart
# bconsole
* status sd

Automatically selected Storage: File1
Connecting to Storage daemon File1 at localhost:9103

fbsd-template.local-sd Version: 7.4.2 (06 June 2016) amd64-portbld-freebsd11.0 freebsd 11.0-RC3
Daemon started 20-Sep-16 21:09. Jobs: run=0, running=0.

Great, now bconsole can connect to the dir and the dir to both the fd and the sd!

Doing a backup

Before finally doing a first test backup, we’re going to edit the director config once more to tweak messaging a bit:

* exit
# vi bacula-dir.conf

Skip down to the Messages resource named Standard and find the following four lines:

mailcommand = "/usr/local/sbin/bsmtp -h localhost -f \"\(Bacula\) \\" -s
\"Bacula: %t %e of %c %l\" %r"
operatorcommand = "/usr/local/sbin/bsmtp -h localhost -f \"\(Bacula\) \\"
-s \"Bacula: Intervention needed for %j\" %r"
mail = root@localhost = all, !skipped
operator = root@localhost = mount

By default, Bacula sends mails if errors occur. We haven’t configured mailing, so this would only result in more errors… Since we’re not building a production environment here, let’s just get rid of this and make life a bit easier while exploring the essential functionality of Bacula. Comment out all four. Then save the file, exit and restart the director once again:

# service bacula-dir restart

Now issue the following commands:

# bconsole
* run

Automatically selected Catalog: MyCatalog
Using Catalog “MyCatalog”
A job name must be specified.
The defined Job resources are:
1: BackupClient1
2: BackupCatalog
3: RestoreFiles
Select Job resource (1-3):

Pick number 1.

Run Backup job
JobName: BackupClient1
Level: Incremental
Client: fbsd-template.local-fd
FileSet: Full Set
Pool: File (From Job resource)
Storage: File1 (From Job resource)
When: 2016-09-20 21:24:13
Priority: 10
OK to run? (yes/mod/no):

Answer “yes”.

Job queued. JobId=1
You have messages.

Ask the director to display the messages:

* mes

You should get a lot of output. If you only got a few lines, you probably were too fast and caught Bacula while it was still working. Give it a second and type mes again. The important line to look for is this one near the end:

Termination: Backup OK

Take a look at all the lines and get a rough idea what has happened here. Do so especially if at this point something bad happened (i.e. the backup failed). In this case check that you did everything as I wrote it here and if that doesn’t help, you’re unfortunately in for some research… But I’ll assume that it worked well and you just did your first Bacula backup. Good work! This means that you’ve also successfully ensured that all daemons can interact with each other.

That’s it for this part of the tutorial. Power down the VM and save another snapshot (when the status has reached poweroff):

* exit
# shutdown -p now
% vagrant status
% vagrant snapshot save tut_2_end

Intermission

We’re still in the early stages of setting up Bacula but at least this time we were able to actually do something by backing up some data. You probably have no idea what you’ve just backed up or where to – but that’s ok for now. We’ll discuss these things in the next part. And in addition to making all daemons able to interact and doing a backup, you’ve done some debugging work. So I think having a break here and talking about backups in more detail later (I also want to cover configuration some more before I do so) makes sense.

I hope that this has been a useful read. Feel free to leave me a comment if you like it (or if you don’t) and of course tell me if I made some mistakes that should be corrected!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s