FreeBSD package management with Pkg (1/2)

FreeBSD is a server operating system popular among more experienced administrators. It’s also often used in appliances or embedded products and it makes a nice desktop, too, if you are a bit more proficient with it. Sure, you don’t read about it all the time. This is because of two things: 1) Linux is usually absorbing most of the attention 2) FreeBSD just silently and reliably does its job. FreeBSD is also the basis of some special-purpose open source operating systems.

Are you new to FreeBSD? You’ve picked a good time to take a look: Version 11.1 has been released recently and a lot of people who have wanted to give it a spin for a while are getting an ISO and try it out. Also OPNsense 17.7, a popular router/firewall OS has been released, also not too long ago.

If you’re coming e.g. from OPNsense, you have a nice WebGUI at your hands that makes controlling and maintaining easy. That does not mean that you wouldn’t benefit from knowing more about how to do package management by hand, however. OPNsense is built on top of FreeBSD which means that you have all the power of that operating system available if you know what to do if the WebGUI doesn’t provide a simple option to do it!

Pkg – where is it?

If you’re using a FreeBSD-based project like OPNsense or a desktop spin like GhostBSD or TrueOS, it’s located in /usr/local/sbin/pkg. With vanilla FreeBSD it’s almost certainly in the same place of course, if you’ve been using the system for a while. What do I mean by that? Well, in the previous post about the history of *nix package management I made the claim that modern FreeBSD comes without a package manager. And that’s actually true! Take a look at the path mentioned above. See what I mean?

FreeBSD uses the directory tree under /usr/local for programs that are installed as add-on packages, i.e. software that is not part of the base system. Yes, that means that even the package manager is a package! How does that work? Well, there’s also /usr/sbin/pkg, which is part of the base system. If pkg is not present on the system, it’s able to bootstrap it as the first package on the system. If the package manager is already installed, it simply acts as a wrapper (thanks to its location and the default PATH variable it takes precedence over the actual pkg). Bootstrapping is as easy as pressing ‘y’, so we don’t really need to cover it beyond this.

Sounds a bit strange, right? Yes, but there’s a good reason for this. Pkg’s life begun as pkg-ng when FreeBSD still used the old pkg_* tools, so it made sense to develop it outside of the base system. But why wasn’t it imported into the base system when the old tools were retired? Actually, FreeBSD maintains ABI stability for the base system for the life-cycle of a release (e.g. 11). The last point release of the 10 branch, FreeBSD 10.4, is just about a month away from now. If pkg had been part of the base, that would mean the new release would have to ship with pkg 1.2 since 1.2.4 was the version available when 10.0 was released! The 11 branch would ship with 1.8, as 1.8.7 was the current version when FreeBSD 11.0 was released.

That would have meant slower progress, since no package would have been allowed to depend on features introduced after 1.2 until 10.4 went EOL – which might be as far away as somewhere near the end of 2019! Fortunately pkg is a package and thus could be improved rapidly: All versions of FreeBSD have pkg 10.1 today instead of 1.8 – or even 1.2.

How to get help?

If you have no idea how to use pkg, you can use pkg help. This will show a list of supported options and commands. And if you want to know even more, you can always type man pkg. There are also man pages for most of the pkg commands, usually named pkg-command. As a shortcut to those, you could also type pkg help command which will display the correct man page.

On the long run it’s not a bad idea to read a bit more about pkg, but if you are just getting started, all the possibilities might overwhelm you. So let’s discuss a few practical examples to get you up to speed easily and quickly, shall we?

Finding packages

As long as you just use pkg to gather information, you can run it as any user. Modifying anything needs root privileges, of course.

By default pkg operates on remote a repository. A repo (common abbreviation) is simply a place where packages are stored and kept accessible together with an index file. To be of any use, pkg needs that index and (with the default configuration) will fetch (or update if it deems the local copy too old) it before performing any action you might want it to do. If you just want to get/update the index, you can run pkg update:

# pkg update
Updating FreeBSD repository catalogue...
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    
Fetching packagesite.txz: 100%    6 MiB 548.9kB/s    00:11    
Processing entries: 100%
FreeBSD repository update completed. 26602 packages processed.
All repositories are up to date.

Provided you can connect to the repo, this will fetch the latest index and process it. Pkg uses a local sqlite database created from the index so that it can quickly and nicely get information from it.

To see if a package in any configured repository is available for installation, use pkg search:

% pkg search bash
bash-4.4.12_2                  GNU Project's Bourne Again SHell
bash-completion-2.5,1          Programmable completion library for Bash
bash-static-4.4.12_2           GNU Project's Bourne Again SHell
bashc-3.2.33.0_1               GNU bash shell extended with visual two-panel file browser
checkbashisms-2.15.10          Check for the presence of bashisms
erlang-mochiweb-basho-2.9.0p2  Erlang library for building lightweight HTTP servers (Basho fork)
mybashburn-1.0.2_4             Ncurses CD burning bash script
p5-Bash-Completion-0.008_1     Extensible system to provide bash completion
p5-Term-Bash-Completion-Generator-0.02.8_1 Generate bash completion scripts

Pkg returns a list of hits, each with package name and version as well as a short comment to give you an idea what the package actually is. If you know either the package name or part of the name, searching is easy enough.

Now let’s assume you are looking for a light-weight web browser but you forgot the name! Since it’s not very likely that the package’s name contains “browser”, how do you search for it? You could simply search the comments e.g. for “web browser”, but that would lead to quite a list. Do you remember anything else about the browser? Let’s say we know that it uses the FLTK toolkit. Let’s see if we can find that package:

% pkg search -c "web browser" | grep -i FLTK                            
dillo-3.0.5                    Fast, small graphical Web browser built upon fltk

There we are, the browser’s name is dillo! If even searching in the comments doesn’t yield what you are searching for, you might even resort to pkg search -D keyword. This will search in the detailed description that each package comes with. Just be prepared for a lot of hits and a wall of text if you’re using common keywords.

Web-based package search

In many cases it’s not a bad idea to use a web-based search. That is what the site FreshPorts provides among other things.

If you’re trying to get into FreeBSD, this is a site that you might want to bookmark. Quite possibly it’ll come in handy rather often. Also spend a little while exploring what it offers. Getting familiar with it is in fact time will spent.

Installing packages

To install a package, just issue pkg install pkgname:

# pkg install chocolate-doom
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 7 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        chocolate-doom: 2.3.0
        sdl_net: 1.2.8_3
        doom-data: 1.0_1
        sdl_mixer: 1.2.12_12
        smpeg: 0.4.4_14
        timidity: 0.2i_1
        libmikmod: 3.3.8

Number of packages to be installed: 7

The process will require 22 MiB more space.
11 MiB to be downloaded.

Proceed with this action? [y/N]:

Just answer y to make pkg fetch and install those packages (any PC without DooM installed just isn’t quite complete after all – I still stick to that even though I haven’t found time to actually fire up that game in years!).

What’s installed?

To query the package database, use pkg info. Without any further arguments this will return a list of all installed packages (so you probably want to pipe it into a pager like less). Searching for a specific package? You can definitely do that using your standard Unix tools, but hold that grep right now! There’s a better way:

% pkg info -x mate-t
ghostbsd-mate-themes-1.4
mate-terminal-1.18.1
mate-themes-3.22.12

The -x switch enables searching with regular expressions. The search in my example shows all packages that contain “mate-t”. You may even want to get used to adding -i, too, which enables case-insensitive search – just in case (there aren’t that many packages in FreeBSD that contain uppercase letters, but some do).

If you found your package, just use pkg info [packagename] (without the square brackets, of course) to get a whole lot of information about the package! But that’s probably more than you wanted and it makes sense to know a few more switches that just give you some specific information.

A nice one is -D. Forget what that post-install message was that told you how to actually get your package to work? This option prints it again:

% pkg info -D chromium
chromium-60.0.3112.101:
Always:
For correct operation, shared memory support has to be enabled
in Chromium by performing the following command as root :

sysctl kern.ipc.shm_allow_removed=1
[...]

Use -d to query information about a package’s dependencies:

% pkg info -d galculator
galculator-2.1.4:
	pango-1.40.6
	gtk3-3.22.15
	gtk-update-icon-cache-2.24.29
	gdk-pixbuf2-2.36.6
	cairo-1.14.8_1,2
	glib-2.50.2_4,1
	gettext-runtime-0.19.8.1_1
	atk-2.24.0

With -l you can list all the files that a package installed in your filesystem:

% pkg info -l mksh
mksh-56_1:
        /usr/local/bin/mksh
        /usr/local/man/man1/mksh.1.gz
        /usr/local/share/examples/mksh/dot.mkshrc
        /usr/local/share/licenses/mksh-56_1/ISCL
        /usr/local/share/licenses/mksh-56_1/LICENSE
        /usr/local/share/licenses/mksh-56_1/ML
        /usr/local/share/licenses/mksh-56_1/catalog.mk

Deleting packages

The last one of the basic operations is removing packages. Here in my example I spotted a package called “tracker”, a filesystem indexer that comes with GNOME. If you don’t use that desktop and find it on your system: Kill it with fire! Nuke it from the system with pkg delete:

# pkg delete tracker
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 2 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
        tracker-1.6.1_9
        nautilus-3.18.5

Number of packages to be removed: 2

The operation will free 21 MiB.

Proceed with deinstalling packages? [y/N]: y

Deinstalling this package will also get rid of Nautilus as well (since tracker is a dependency for that). Since the GNOME team pretty much ruined that once decent file manager, I’m not going to shed any tears over that loss and press enter. Pkg will then delete all the files associated with the package and un-register it in the package database.

What’s next?

That’s it for the basics in my opinion. The next post will show off a few of the more advanced features that Pkg offers.

Advertisements

The history of *nix package management

Very few people will argue against the statement that Unix-like operating systems conquered the (professional) world due to a whole lot of strong points – one of which is package management. Whenever you take a look at another *nix OS or even just another Linux distro, one of the first things (if not the first!) is to get familiar with how package management works there. You want to be able to install and uninstall programs after all, right?

If you’re looking for another article on using jails on a custom-built OPNsense BSD router, please bear with me. We’re getting there. To make our jails useful we will use packages. And while you can safely expect any BSD or Linux user to understand that topic pretty well, products like OPNsense are also popular with people who are Windows users. So while this is not exactly a follow-up article on the BSD router series, I’m working towards it. Should you not care for how that package management stuff all came to be, just skip this post.

When there’s no package manager

There’s this myth that Slackware Linux has no package manager, which is not true. However Slackware’s package management lacks automatic dependency resolving. That’s a very different thing but probably the reason for the confusion. But what is package management and what is dependency resolving? We’ll get to that in a minute.

To be honest, it’s not very likely today to encounter a *nix system that doesn’t provide some form of package manager. If you have such a system at hand, you’re quite probably doing Linux from Scratch (a “distribution” meant to learn the nuts and bolts of Linux systems by building everything yourself) or have manually installed a Linux system and deliberately left out the package manager. Both are special cases. Well, or you have a fresh install of FreeBSD. But we’ll talk about FreeBSD’s modern package manager in detail in the next post.

Even Microsoft has included Pkgmgr.exe since Windows Vista. While it goes by the name of “package manager”, it turns pale when compared to *nix package managers. It is a command-line tool that allows to install and uninstall packages, yes. But those are limited to operating system fixes and components from Microsoft. Nice try, but what Redmond offered in late 2006 is vastly inferior to what the *nix world had more than 10 years earlier.

There’s the somewhat popular Chocolatey package manager for Windows and Microsoft said that they’d finally include a package manager called “one-get” (apt-get anyone?) with Windows 10 (or was it “nu-get” or something?). I haven’t read a lot about it on major tech sites, though, and thus have no idea if people are actually using it and if it’s worth to try out (I would, but I disagree with Microsoft’s EULA and thus I haven’t had a Windows PC in roughly 10 years).

But how on earth are you expected to work with a *nix system when you cannot install any packages?

Before package managers: Make magic

Unix begun its life as an OS by programmers for programmers. Want to use a program on your box that is not part of your OS? Go get the source, compile and link it and then copy the executable to /usr/local/whatever. In times where you would have just some 100 MB of storage in total (or even less), this probably worked well enough. You simply couldn’t go rampage and install unneeded software anyways, and sticking to the /usr/local scheme you separate optional stuff from the actual operating system.

More space became available however and software grew bigger and more complex. Unix got the ability to use libraries (“shared objects”), ELF executables, etc. To solve the task of building more complicated software easily, make was developed: A tool that read a Makefile which told it exactly what to do. Software begun shipping not just with the source code but also with Makefiles. Provided that all dependencies existed on the system, it was quite simple to build software again.

Compilation process (invoked by make)

Makefiles also provide a facility called “targets” which made a single file support multiple actions. In addition to a simple make statement that builds the program, it became common to add a target that allowed for make install to copy the program files into their assumed place in the filesystem. Doing an update meant building a newer version and simply overwriting the files in place.

Make can do a lot more, though. Faster recompiles by to looking at the generated file’s timestamp (and only rebuilding what has changed and needs to be rebuilt) and other features like this are not of particular interest for our topic. But they certainly helped with the quick adoption of make by most programmers. So the outcome for us is that we use Makefiles instead of compile scripts.

Dependency and portability trouble

Being able to rely on make to build (and install) software is much better than always having to invoke compiler, linker, etc. by hand. But that didn’t mean that you could just type “make” on your system and expect it to work! You had to read the readme file first (which is still a good idea, BTW) to find out which dependencies you had to install beforehand. If those were not available, the compilation process would fail. And there was more trouble: Different implementations of core functionality in various operating systems made it next to impossible for the programmers to make their software work on multiple Unices. Introduction of the POSIX standard helped quite a bit but still operating systems had differences to take into account.

Configure script running

Two of the answers to the dependency and portability problems were autoconf and metaconf (the latter is still used for building Perl where it originated). Autoconf is a tool used to generate configure scripts. Such a script is run first after extracting the source tarball to inspect your operating system. It will check if all the needed dependencies are present and if core OS functionality meets the expectations of the software that is going to be built. This is a very complex matter – but thanks to the people who invested that tremendous effort in building those tools, actually building fairly portable software became much, much easier!

How to get rid of software?

Back to make. So we’re now in the pleasant situation that it’s quite easy to build software (at least when you compare it to the dark days of the past). But what would you do if you want to get rid of some program that you installed previously? Your best bet might be to look closely at what make install did and remove all the files that it installed. For simple programs this is probably not that bad but for bigger software it becomes quite a pain.

Some programs also came with an uninstall target for make however, which would delete all installed files again. That’s quite nice, but there’s a problem: After building and installing a program you would probably delete the source code. And having to unpack the sources again to uninstall the software is quite some effort if you didn’t keep it around. Especially since you probably need the source for exactly the same version as newer versions might install more or other files, too!

This is the point where package management comes to the rescue.

Simple package management

So how does package management work? Well, let’s look at packages first. Imagine you just built version 1.0.2 of the program foo. You probably ran ./configure and then make. The compilation process succeeded and you could now issue make install to install the program on your system. The package building process is somewhat similar – the biggest difference is that the install destination was changed! Thanks to the modifications, make wouldn’t put the executable into /usr/local/bin, the manpages into /usr/local/man, etc. Instead make would then put the binaries e.g. into the directory /usr/obj/foo-1.0.2/usr/local/bin and the manpages into /usr/obj/foo-1.0.2/usr/local/man.

Installing tmux with installpkg (on Slackware)

Since this location is not in the system’s PATH, it’s not of much use on this machine. But we wanted to create a package and not just install the software, right? As a next step, the contents of /usr/obj/foo-1.0.2/ could be packaged up nicely into a tarball. Now if you distribute that tarball to other systems running the same OS version, you can simply untar the contents to / and achieve the same result as running make install after an unmodified build. The benefit is obvious: You don’t have to compile the program on each and every machine!

So far for primitive package usage. Advancing to actual package management, you would include a list of files and some metadata into the tarball. Then you wouldn’t extract packages by hand but leave that to the package manager. Why? Because it would not only extract all the needed files. It will also record the installation in its package database and keep the file list around in case it’s needed again.

Uninstalling tmux and extracting the package to look inside

Installing using a package manager means that you can query it for a list of installed packages on a system. This is much more convenient than ls /usr/local, especially if you want to know which version of some package is installed! And since the package manager keeps the list of files installed by a package around, it can also take care of a clean uninstall without leaving you wondering if you missed something when you deleted stuff manually. Oh, and it will be able to lend you a hand in upgrading software, too!

That’s about what Slackware’s package management does: It enables you to install, uninstall and update packages. Period.

Dependency tracking

But what about programs that require dependencies to run? If you install them from a package you never ran configure and thus might not have the dependency installed, right? Right. In that case the program won’t run. As simple as that. This is the time to ldd the program executable to get a list of all libraries it is dynamically linked against. Note which ones are missing on your system, find out which other packages provide them and install those, too.

Pacman (Arch Linux) handles dependencies automatically

If you know your way around this works ok. If not… Well, while there are a lot of libraries where you can guess from the name which packages they would likely belong to, there are others, too. Happy hunting! Got frustrated already? Keep saying to yourself that you’re learning fast the hard way. This might ease the pain. Or go and use a package management system that provides dependency handling!

Here’s an example: You want to install BASH on a *nix system that just provides the old bourne shell (/bin/sh). The package manager will look at the packaging information and see: BASH requires readline to be installed. Then the package manager will look at the package information for that package and find out: Readline requires ncurses to be present. Finally it will look at the ncurses package and nod: No further dependencies. It will then offer you to install ncurses, readline and BASH for you. Much easier, eh?

Xterm and all dependencies downloaded and installed (Arch Linux)

First package managers

A lot of people claim that the RedHat Package Manager (RPM) and Debian’s dpkg are examples of the earliest package managers. While both of them are so old that using them directly is in fact inconvenient enough to justify the existence of another program that allows to use them indirectly (yum/dnf and e.g. apt-get), this is not true.

PMS (short for “package management system”) is generally regarded to be the first (albeit primitive) package manager. Version 1.0 was ready in mid 1994 and used on the Bogus Linux distribution. With a few intermediate steps this lead to the first incarnation of RPM, Red Hat’s well-known package manager which first shipped with Red Hat Linux 2.0 in late 1995.

FreeBSD 1.0 (released in late 1993) already came with what is called the ports tree: A very convenient package building framework using make. It included version 0.5 of pkg_install, the pkg_* tools that would later become part of the OS! I’ll cover the ports tree in some detail in a later article because it’s still used to build packages on FreeBSD today.

Part of a Makefile (actually for a FreeBSD port)

Version 2.0-RELEASE (late 1994) shipped the pkg_* tools. They consisted of a set of tools like pkg_add to install a package, pkg_info to show installed packages, pkg_delete to delete packages and pkg_create to create packages.

FreeBSD’s pkg_add got support for using remote repositories in version 3.1-RELEASE (early 1999). But those tools were really showing their age when they were put to rest with 10.0-RELEASE (early 2014). A replacement has been developed in form of the much more modern solution initially called pkg-ng or simply pkg. Again that will be covered in another post (the next one actually).

With the ports tree FreeBSD undoubtedly had the most sophisticated package building framework of that time. Still it’s one of the most flexible ones and a bliss to work with compared to creating DEB or RPM packages… And since Bogus’s PMS was started at least a month after pkg_install, it’s even entirely possible that the first working package management tool was in fact another FreeBSD innovation.