This is the third part of my Bacula tutorial. The first part covered some basics as well as installing Bacula and starting the three daemons that it consists of. Part two dealt with modifying the default configuration files in a way that allowed all components of Bacula to interact with each other locally. And finally a test backup was done (without knowing details like what exactly would even be backed up!) just to ensure that communication between the daemons really works.
In this post we’ll clean up and split the configuration files to make them more comprehensible. We’ll create our own fileset, add a new device and assign that to a new storage resource. We’ll also customize a backup job on the bconsole to do a new backup using our self-created resources.
The third part continues where the second one left off. During this tutorial series we use VirtualBox VMs managed by Vagrant to make things as simple as possible. If you don’t know how to use them, have a look here and here for a detailed introduction into using Vagrant and on how to prepare a VM template with FreeBSD that is used in the Bacula tutorial.
Configuration cleanup
Let’s jump right into it! So restore the snapshot from the previous post, SSH into the machine, become root and change into Bacula’s configuration directory:
% cd ~/vagrant/backuphost
% vagrant snapshot restore tut_2_end
% vagrant ssh
% sudo su -
# cd /usr/local/etc/bacula/
Ok, here we are. Before we can do anything actually somewhat useful, we need to sort out things and get an idea of how the configuration works (what all those resources do, etc). Bacula’s default configuration is rather huge which doesn’t exactly make this an easy task. But there’s an easy solution to it: Let’s start by deleting each and every line that’s commented out in all of the configuration files!
Yes, some of these lines provide useful information but that’s not really lost:
# ls -1
bacula-barcodes
bacula-barcodes.sample
bacula-dir.conf
bacula-dir.conf.sample
bacula-fd.conf
bacula-fd.conf.sample
bacula-sd.conf
bacula-sd.conf.sample
bconsole.conf
bconsole.conf.sample
As you can see, there are files like bacula-dir.conf.sample, bacula-fd.conf.sample, and so on. These are the unaltered default configuration files and you can always look at those as a reference. So before we move on: Edit the four config files that we’ve worked with so far (bacula-dir.conf, bacula-fd.conf, bacula-sd.conf and bconsole.conf) and get rid of all the commented out lines.
Removing these lines (and a few empty ones that were between them and didn’t have any purpose on their own) shrinks the configuration down from over 600 lines to less than 300. While that’s much better, it’s still a lot. Let’s throw more stuff away, shall we? There’s a “restricted director” that’s used for monitoring. We’re not going to use that here so all those resources with the name of fbsd-template.local-mon can be deleted from all three daemon’s config files. Do that now.
At this point restart all three daemons (and check if they are running afterwards) to make sure the configuration is still valid:
# service bacula-dir restart
# service bacula-fd restart
# service bacula-sd restart
The files bconsole.conf and bacula-fd.conf are trivial now with less than 20 lines each. Still bacula-sd.conf with about 80 lines is barely acceptable and especially bacula-dir.conf with over 170 remains too big for my taste!
Configuration splitting
Of course we cannot simply go rampage and delete more and more resources from the director configuration. So how to make it smaller (while maintaining the functionality)? Bacula supports breaking up larger configuration files. To do so we put some resources into their own file and include that in the main config file! Bacula doesn’t care in which files you stuff its resources and it also doesn’t really care about the order. But we do that to make things easier to maintain and so it makes sense to think about what resources should be grouped together.
To keep things organized, it’s best to create a separate directory for includes (or even multiple directories if you have complex configuration and need to separate things further). Choose a name that fits your taste:
# mkdir includes
Here’s my suggestion (you might want to do this completely differently – just do what makes sense for you) for splitting bacula-dir.conf:
- Move all JobDefs and Job resources into includes/dir_job.conf
- Move all FileSet resources into includes/dir_fileset.conf
- Move all Schedule resources into includes/dir_schedule.conf
- Move all Storage resources into includes/dir_storage.conf
- Move all Pool resources into includes/dir_pool.conf
Now let’s include those configuration files that we split off! We could do so by including them one by one by adding the following lines at the end of bacula-dir.conf like this:
@/usr/local/etc/bacula/includes/dir_jobs.conf
@/usr/local/etc/bacula/includes/dir_filesets.conf
@/usr/local/etc/bacula/includes/dir_schedule.conf
@/usr/local/etc/bacula/includes/dir_storage.conf
@/usr/local/etc/bacula/includes/dir_pool.conf
That would work (just make sure you use absolute paths to save you some potential headaches). But don’t put it in there (or remove it again if you already have)! There’s a better way of doing it; just add the one following line instead:
@|"sh -c 'for f in /usr/local/etc/bacula/includes/dir_*.conf ; do echo @${f} ; done'"
This will use the shell to automatically include all files in /usr/local/etc/bacula/includes/ that start with dir_ and end in .conf. Much better, eh? And it has the advantage of automatically pulling in includes/dir_messages.conf or something like that should you ever decide to split e.g. the Messages resources from the main dir config file as well.
Now we have a bunch of files with nicely separated resource types and the biggest ones are around 40 lines long. Much more comprehensible if you ask me. Let’s quickly split up the sd config as well by just putting all the Device resources into include/sd_device.conf and including it adding the following line into bacula-sd.conf:
@|"sh -c 'for f in /usr/local/etc/bacula/includes/sd_*.conf ; do echo @${f} ; done'"
Restart dir and sd again to see if the configuration is still ok:
# service bacula-dir restart
# service bacula-sd restart
Hostname
We’ve given our base box a hostname of “fbsd-template.local”. That’s fine for a template – but this machine is not a template anymore. We’ll spin up more VMs from the same base box (template) later so it makes sense to do some proper naming. We could just change the hostname and Bacula would not care at all here since a resource’s name is just a name. But for human operators things can get pretty confusing pretty fast – and that’s what we wanted to avoid, right? So let’s edit the four config files now and change the Name of the various resources from e.g. fbsd-template.local-dir to backuphost.local-dir. Once you’ve replaced it nine times in the actual config files, you’re done here.
But we must not forget to also change it twice in includes/dir_job.conf or the director won’s start since the jobs refer to the old name! And while we’re at it, let’s also change the first Job resource’s name in the same file from BackupClient1 to Backuphost.local. Save the file and exit the editor.
Now let’s change the hostname that FreeBSD will assign itself during startup. To do so we need to edit rc.conf:
# vi /etc/rc.conf
Change the corresponding line so that it holds the new hostname:
hostname="backuphost.local"
Now is an excellent time to take another snapshot (so you can restore that if something goes wrong and don’t have to redo all the config splitting, etc.) and since we’ve changed the hostname we need to reboot anyways. So let’s shutdown the VM, snapshot and start the VM again so that it comes back up with the new hostname:
# shutdown -p now
% vagrant status
% vagrant snapshot save tut_3_mid
% vagrant up
Let’s SSH back into it, switch to root and see if the hostname changed:
% vagrant ssh
% sudo su -
# hostname
backuphost.local
That looks good. Also all Bacula services should be running (feel free to double-check) so we can move on.
What to back up: Filesets
Let’s talk about FileSets next. We’ve put definitions of this kind of resources into their own configuration file that’s then included into the dir configuration. Now we want to add a new fileset. We do so by editing:
# cd /usr/local/etc/bacula
# vi includes/dir_fileset.conf
Just add the following lines to it:
FileSet {
Name = "etc"
Include {
Options {
signature = MD5
}
File = /etc
File = /usr/local/etc
}
Exclude {
File = /etc/casper
}
}
What does this do? It defines another fileset named “etc” which includes the directory /etc. By default Bacula will recursively include files and subdirectories when it is told to back up a directory. So this fileset will effectively backup almost all of the system’s configuration files (which reside in /etc and /usr/local/etc).
Let’s assume we don’t use Capsicum (FreeBSD’s sandboxing framework in case you’ve never heard of it) and don’t want to back up configuration for the Casper daemon (which is a support daemon for Capsicum). The above example excludes /etc/casper from being backed up.
And then there’s the Signature directive. It tells Bacula to create a signature for each file it backs up using the MD5 algorithm. While it is not necessary it is strongly recommended to let Bacula calculate signatures (checksums). Bacula uses these to find out if a file has been altered since it was last backed up even if e.g. the file size didn’t change. Even though it takes a little more space if you use signatures, doing so totally makes sense.
Save the file and exit. You don’t have to restart the director, yet, because we’re going to make one more change.
Where to back up to: Storage
Let’s say we want to store our backups in /var/backup. To do this, we first create the directory and make sure Bacula can access it:
# mkdir /var/backup
# chown bacula:bacula /var/backup
Now we have to inform the sd of the new “device”. Edit the configuration:
# vi includes/sd_device.conf
and add the following lines:
Device {
Name = Var-File
Media Type = File3
Archive Device = /var/backup
LabelMedia = yes;
Random Access = Yes;
AutomaticMount = yes;
RemovableMedia = no;
AlwaysOpen = no;
Maximum Concurrent Jobs = 5
}
Here we create a new “device” for Bacula to use – this “device” is in our case just a directory (/var/backup) on the disk. It is important that you use unique media types; since there’s already File1 and File2 in the default configuration, we go with File3 here. In fact you could choose any unique name that makes some sense to you. Ignore the other settings (or look them up in the Bacula documentation if you want to know more). Save the file and exit. However you won’t be able to use it, yet, as the dir does not know anything about it. So we need to edit its configuration as well:
vi includes/dir_storage.conf
and add the following lines:
Storage {
Name = File3
Address = localhost
SDPort = 9103
Password = "sdPASSWORD"
Device = Var-File
Media Type = File3
Maximum Concurrent Jobs = 10
}
This creates a new storage that can hold backups. It is defined for the director who will send it to the fd if it initiates a backup. Save and exit. Now we need to restart dir and sd to load the new configuration:
# service bacula-dir restart
# service bacula-sd restart
Backup time!
We’ve made a lot of changes to the configuration so let’s attempt to do another backup:
# bconsole
* run
Automatically selected Catalog: MyCatalog
Using Catalog “MyCatalog”
A job name must be specified.
The defined Job resources are:
1: Backuphost.local
2: BackupCatalog
3: RestoreFiles
Select Job resource (1-3):
Select the first one:
Run Backup job
JobName: Backuphost.local
Level: Incremental
Client: backuphost.local-fd
FileSet: Full Set
Pool: File (From Job resource)
Storage: File1 (From Job resource)
When: 2016-09-23 18:36:24
Priority: 10
OK to run? (yes/mod/no):
Choose mod this time:
Parameters to modify:
1: Level
2: Storage
3: Job
4: FileSet
5: Client
6: When
7: Priority
8: Pool
9: Plugin Options
Select parameter to modify (1-9):
Select 4:
The defined FileSet resources are:
1: Full Set
2: Catalog
3: etc
Select FileSet resource (1-3):
Of course we want to use 3, then mod and then 2:
The defined Storage resources are:
1: File1
2: File2
3: File3
Select Storage resource (1-3):
Choose 3, then mod and then 8:
The defined Pool resources are:
1: Default
2: File
3: Scratch
Select Pool resource (1-3):
Go with the first one:
Run Backup job
JobName: Backuphost.local
Level: Incremental
Client: backuphost.local-fd
FileSet: etc
Pool: Default (From User input)
Storage: File3 (From user selection)
When: 2016-09-23 18:38:24
Priority: 10
OK to run? (yes/mod/no):
Now type in yes:
Job queued. JobId=2
You have messages.
Let’s see how that went:
* mes
[…]
23-Sep 18:41 backuphost.local-dir JobId 2: Using Device “Var-File” to write.
23-Sep 18:41 backuphost.local-sd JobId 2: Job Backuphost.local.2016-09-23_18.41.29_03 is waiting. Cannot find any appendable volumes.
Please use the “label” command to create a new Volume for:
Storage: “Var-File” (/var/backup)
Pool: Default
Media type: File3
Oh my! What’s happening now again? We need to create a label first before Bacula can use the new storage. Let’s just do that now and talk about the detail later (there’s a bit more to storage actually):
* label
The defined Storage resources are:
1: File1
2: File2
3: File3
Select Storage resource (1-3):
Select target 3 of course. Bacula will then ask you to give that volume a name:
Enter new Volume name:
Let’s just call it file3a:
Defined Pools:
1: Default
2: File
3: Scratch
Select the Pool (1-3):
Select 1:
Connecting to Storage daemon File3 at localhost:9103 …
Sending label command for Volume “file3a” Slot 0 …
3000 OK label. VolBytes=212 VolABytes=0 VolType=1 Volume=”file3a” Device=”Var-File” (/var/backup)
Catalog record for Volume “file3a”, Slot 0 successfully created.
Requesting to mount Var-File …
3001 OK mount requested. Device=”Var-File” (/var/backup)
Seems like everything is ok. Wait a few seconds and see if the job that has been waiting in line has been run automatically:
* mes
[ … quite some output … ]
Again look especially for the most important line. If you can find this one:
Termination: Backup OK
everything has worked out.
Let’s save our progress by shutting down the VM and taking the next snapshot:
* exit
# shutdown -p now
% vagrant status
% vagrant snapshot save tut_3_end
Intermission
Alright! We just did our second backup with Bacula and this time we actually even know what we did there. While it’s not perfect to modify the backup job like this we’re certainly making progress (and it makes sense to have done it like that at least once). We’ll take care of the job defaults, discuss labels, volumes, etc. next time and also do a restore. But for now take a break. You’ve earned it.
Got any comments or figured that there’s something wrong with what I wrote? Just drop me a line in the comments.
eerielinux 