FreeBSD: Building software from ports (2/2)

My previous post discussed what ports are, where they can be found on FreeBSD and what the files of which a port is composed of look like. This post will now detail how to use ports to build software on FreeBSD (the other BSDs have ports trees that work somewhat similar but are not identical. There are important differences!).

Packages and ports: A word of warning

The ports system works hand in hand with FreeBSD’s package manager Pkg. It makes little difference if some software on your machine was installed via a package or directly from ports – packages are in fact actually built from ports! Still it is not really recommended to mix packages and ports. In past times it was strongly discouraged. Things have changed since then. I’ve done it a lot – and mostly got away with it. Don’t rely on it, though, especially if you’re new to the whole topic. Feel free to do it on a test system and be completely happy – or face subtle and annoying breakage. You cannot know up front.

What’s the deal here? Modern software is a complex thing. Most programs rely on other programs or external libraries. A lot of programs can be configured at run time in certain ways. There are however decisions about program functionality that have to be made at compile time. The ports system allows you to build software with compile-time options other than the default. Pre-compiled packages have no chance to know that you choose to deactivate an option when you built a library yourself that they make use of. They assume that this feature is present (it was available on the system the package was built on after all!). And what can one poor program do in that case? Crash, explode, malfunction… A lot of things.

And then there’s the problem of mixing versions which can lead to all kinds of fun. If you stick with either ports or packages, you always have a consistent system with versions that are known to play together well (as long as the maintainers do their job well – we’re all humans and errors do occur).

Just keep that in mind when thinking about mixing programs installed from packages and ports on one system. You can do that. But it doesn’t mean you should. Enabling more options is generally safer than removing ones set by default. It can still have consequences. This is Unix though. Do whatever you see fit – and claim the responsibility. Your choice.

Most basic ports building

Building a software from ports is extremely easy. Go to the directory of a port and type make. Yes, that’s all! Let’s assume the port has no unsatisfied dependencies. The ports system will then check to see if the source code tarball is present in /usr/ports/distfiles. If it isn’t, it will automatically download it. Then it extracts the source code, prepares everything for the compilation and compiles it.

Building the ‘pkg’ port

On my fresh example system I build the Pkg manager from ports first – it’s needed for every other port anyway. Once everything has finished I get my shell back.

Building of Pkg completed

Installing the program is just as easy: Use make install

Installing the newly built port

That’s it, Pkg is now installed. We’re basically done with that port. However there’s still the “work” directory left over from the building process. To tidy up our port’s directory we can issue make clean.

Cleaning up after the build

Dependency handling

On to a just slightly more complex example. I want to build and install an old version of the LUA interpreter which depends on another port, libedit. Of course I could build devel/libedit first and then lang/lua51. In that case it wouldn’t be so bad. But if you think of larger programs with hundreds of dependencies that approach would be a nightmare.

So what to do about it? Well, nothing actually. The ports system takes care of it automatically! Just have it build LUA and it will figure out that it has to build the dependency first.

Building, installing and cleaning up in one command

The parameters to make that we used above are called make targets, BTW, and can be combined. That means it’s perfectly fine to issue make install clean together as you can see in the picture above.

Dependencies are handled automatically

The clean make target is also applied to all ports that were built as a dependency for the current port. Things like this make ports very convenient to use.

More on make and targets

Make targets can depend on other make targets. When you issue make install these are the targets that are actually run:

  • make config (more on that in a minute)
  • make fetch (fetch all files needed to build the port)
  • make checksum (check integrity of downloaded file(s))
  • make depends (check for missing dependencies and build/install those)
  • make extract (extract distfile(s) for the port)
  • make patch (apply patches for this port, if any)
  • make build (actually build the port)
  • make install (install the newly-built program)

If you type make checksum for example, all targets up to and including that one will run (that is config, fetch and checksum in that case). Running just make without any target will assume the default target which is equivalent to make build.

Also make will take an argument to look for the Makefile in another directory if you wish. So instead of doing e.g. this:

# cd /usr/ports/archivers/bzip2
# make install clean

you could also simply do this:

# make -C /usr/ports/archivers/bzip2 install clean

You’re in control: Ports options

So far it’s all nice and well but there’s no real advantage to using ports instead of packages. May I introduce ports options? Let’s say you we want to build BASH. If issue make in shells/bash, this is what happens:

Build options for BASH

The port ports-mgmt/dialog4ports is fetched and installed. It’s so small that you might miss it but it’s quite important. It’s needed to display the menu in the picture above which lets you set various options for the port.

You can now e.g. choose to not install the documentation if you’re short on space on a small or embedded system (sure, you wouldn’t actually compile on such a system, but that’s only an example, right?). If you don’t want BASH to support any foreign languages, deselect NLS. In case you feel that BASH’s built-in help is useless (did you ever issue the help command when you ran BASH?), you can cut that feature. Things like that.

If you see the option configuration for a port the first time, you see the default configuration. In general it’s a good idea to leave options alone if you’re in doubt what they do (do a little research if you have the time). Of course you’re also free to experiment with them. It’s your system.

Once you’re happy, accept your selection and the source tarball is being fetched, extracted, etc. You know the score.

Build options for bison

But what’s that? Another configuration menu (for bison)? And another (m4) and another (texinfo), etc… It’s 8 menus for a rather basic program like BASH! And worse: The building process will run and build dependencies and when a port with options is reached, the process is interrupted and prompts the user.

Now imagine you’re building a whole graphical desktop like MATE… Currently even the basic desktop would build no less that 338 dependency packages on a fresh system! And there’s quite a few ports on the list which build rather heavy software that takes it’s time compiling. It would totally make sense to let it build over night or at least not require you to keep staring at the screen, waiting for the next options selection to confirm, right?

Recursive operations

Exactly that’s why recursive operations are supported by the ports system. The standard make target that was implicitly run to open the options dialog is make config. The recursive option which would run the same on each and every port that’s listed as a dependency for the current port is make config-recursive.

If you want to build MATE as mentioned in the previous example, that would start a true marathon of options for you to configure. However it’s still a lot better to be able to do this up front so that the build process can run uninterruptedly afterwards.

Oh, and don’t be surprised if you went through it all only to find that still another configuration dialog pops up later! Why? Most likely you enabled an option on some package that made it depend on another package that’s not a dependency by default. And that package may need to have its options configured, too. So if you changed any options it makes sense to run make config-recursive again until no more new option dialog windows are displayed!

Recursively fetching distfiles for security/sudo

You can also do make fetch-recursive to fetch the distfiles for the current port and all dependencies. Again: Keep in mind that enabling more options may lead to new dependencies. If you want to make sure that you have all the distfiles, you might want to run make fetch-recursive again after changing ports options.

Other things to know

Wonder where the all the options are saved? They are stored in text files in /var/db/ports/category_portname. But there’s no need to edit or delete them; if you want to get rid of them, there’s make rmconfig to do that. Also make rmconfig-recursive exists if you feel like blowing away a huge amount of them.

Ports options in /var/db/ports

Another thing that comes in handy is make build-depends-list which will show you a list of ports that will be built as build dependencies for your current port. If you want to see the runtime dependencies you would use make run-depends-list. And then there’s also make all-depends-list which will show you each and every port that would be installed if you chose to build the current port.

Showing port dependencies

You should also know that you can deinstall a port by using make deinstall. Yes, it is also possible to remove the package using pkg delete but that will lead to a problem. The ports infrastructure keeps track of installed ports separately and Pkg does not know anything about this. So even if your package is removed, the Ports infrastructure will insist that it is still installed and there’s something very wrong with your system!

Now what to do if you have that case? Use make reinstall to install the package again even though ports thinks that it’s already installed.

More on ports?

To be honest, there’s quite a bit more to ports than I could cover here. You may want to man 7 ports to see what other targets are available and what they do. Also we haven’t even touched how to keep your system updated when using ports!

The ports infrastructure is a great means of installing customized programs on your system. It’s quite easy to use as you’ve seen. But things can be made even easier – which is why there are helper tools available. I will write a follow-up article covering those (not the next one, though). But for now enjoy all of those new possibilities with software on your FreeBSD machines!


FreeBSD: Building software from ports (1/2)

In my previous two (link) posts (link) I wrote about using Pkg, FreeBSD’s package manager.

Pre-built binary packages are convenient to use but sometimes you need some more flexibility, want an application that cannot not be distributed in binary form due to license issues (or have some other requirements). Building software by hand is certainly possible – but with all the things involved, this can be a rather tedious process. It’s also slow, error-prone and there’s often no clean way to get rid of that stuff again. FreeBSD Ports to the rescue!

This first part is meant as a soft introduction to FreeBSD’s ports, assuming no prior knowledge (if I fail to explain something, feel free to comment on this post). It will give you enough background information to understand ports enough to start using them in the next article.

What “Ports” are

When programmers talk about porting something over, what they originally meant is this: Take an application that was written with one processor architecture in mind (say i386) and modify the source so that it runs on another (arm64 for example) afterwards. The term “porting” is also used when modifying the source of any program to make it run on another OS. The version that runs on the other architecture/OS is called a port of the original program to a different platform.

FreeBSD uses the term slightly differently. There’s a lot of software written e.g. for Linux that will build and work on FreeBSD just fine as it is. Even though it does not require any changes, that software might be ported to FreeBSD. So in this case “porting” does not mean “make it work at all” but make it easily available. This is done by creating a port for any program. That term doesn’t mean a variant of the source code in this case but rather a means to give you easy access to that software on FreeBSD.

So what is a port in FreeBSD? Actually a port is a directory with a bunch of files in it. The heart of it is one file that basically is a recipe if you will. That recipe contains everything needed to build and install the port (and thus have the application installed on your machine in the end). Following this metaphor you could think of all the ports as a big cookbook. Formally it is known as the Ports collection. All those files in your filesystem related to ports are refered to as the Ports tree.

How to get the Ports tree

There are several options to obtain a copy of the ports tree. When you install FreeBSD you can decide whether or not to install it, too. I usually don’t do that because on systems that use binary packages only. It wastes only about 300 MB of space, but more importantly consists of almost 170.000 files (watch your inodes on embedded devices!). Take a look at /usr/ports: If that directory is empty your system is currently missing the ports tree.

The simplest way to get it is by using portsnap:

# portsnap fetch extract

If you want to update the tree later, you can use:

# portsnap fetch update

Another way is to use Subversion. This is more flexible: With portsnap you always get the current tree while Subversion also allows you to checkout older revisions, too. If you plan to become a ports developer, you will probably want to use Subversion for tools like svndiff. If you just want to use ports, portsnap should actually suffice. All currently supported versions of FreeBSD contain a light-weight version of Subversion called svnlite.

Here’s how to checkout the latest tree:

# svnlite checkout /usr/ports

If you want to update it later run:

# svnlite update /usr/ports

Old versions of the tree

You normally shouldn’t need these but it’s good to know that they exist. Using Subversion you can also retrieve old trees. Be sure that /usr/ports is empty (including for Subversion’s dot directories) or Subversion will see that there’s already something there and won’t do the checkout. If for example you want the ports tree as it existed in 2016Q4, you can retrieve it like this:

# svnlite checkout /usr/ports

There are also several tags available that allow to get certain trees. Maybe you want to see which ports were available when FreeBSD 9.2 was released. Get the tree like this:

# svnlite checkout /usr/ports

And if you need the last tree that is guaranteed to work with 9.x there’s another special tag for it:

# svnlite checkout /usr/ports

Keep in mind though that using old trees is risky because they contain program versions with vulnerabilities that have since been found! Also mind that it’s NOT a smart thing to simply get the tree for RELEASE_7_EOL because it still holds a port for PHP 5.2 and you thought that it would be cool to offer your customers as many versions as possible. Yes, it may be possible that you can still build it if you invest some manual work. But no, that doesn’t make it a good idea at all.

Oh, and don’t assume that old ports trees will be of any use on modern versions of FreeBSD! The ports architecture changed quite a bit over time, the most notable change being the replacement of the old pkg_* tools with the new Pkg. Ports older than a certain time definitely won’t build in their old, unmodified state today (and I say it again: You really shouldn’t bother unless you have a very special case).

Port organization

Take a look at the contents of /usr/ports on a system that has the tree installed. You will find over 60 directories there. There are a few special ones like distfiles (where tarballs with program’s source code get stored – might be missing initially) or Mk that holds include files for the ports infrastructure. The others are categories.

If you’re looking for a port for Firefox, that will be in www. GIMP is in graphics and it’s probably no surprise that Audacious (a music player) can be found in audio. Some program’s categories will be less obvious. LibreOffice is in editors which is not so bad. But help2man for example is in misc and not in converters or devel as at least I would expect if I didn’t know. In general however after a while of working with ports you will have a pretty good chance to guess where things are.

Say we are interested in the port for the window manager Sawfish for example. It’s located in /usr/ports/x11-wm/sawfish. Let’s take a closer look at that location and take it apart:

/usr/ports is the “ports directory”.
x11-wm (short for X11 window managers) is the category.
sawfish is the individual port’s name.

When referring to where a port lives, you can omit the ports directory since everybody is assumed to know where it is. The important information when identifying a port is the category and the name. Together those form what is known as the port origin (x11-wm/sawfish in our case).

How to find a port in the tree

There are multiple methods to find out the origin for the port you are looking for. Probably the simplest one is using whereis. If we didn’t know that sawfish is in x11-wm/sawfish we could do this:

% whereis sawfish
sawfish: /usr/ports/x11-wm/sawfish

This does however only work if you know the exact name of the port. And there’s a little more to it: Sometimes the name of a port and a package differ! This is often the case for Python-based packages. I have SaltStack installed, for example. It’s a package called py27-salt:

% pkg info -x salt

If we were to look for that, we wouldn’t find it:

% whereis py27-salt

So where is the port for the package?

% pkg info py27-salt
Name           : py27-salt
Origin         : sysutils/py-salt

Here you can see that the port’s name is py-salt! The “27” gets added when the package is created and reflects the version of Python that it’s build against. You may also see some py3-xyz ports. In those cases the name reflects that the port cannot be built with Python 2.x. The package will still be called py36-xyz, though (or whatever the default Python 3.x version is at that time)!

When discussing package management I recommended FreshPorts and when working with ports it can be useful, too. Search for some program’s name and it might be easier for you to find the package name and the port origin for it!

What a port looks like

Let’s take a look at the port for the zstd compression utility:

% ls /usr/ports/archivers/zstd/
distinfo	Makefile	pkg-descr	pkg-plist

So what have we here? The simplest file is pkg-descr. Each package has a short and a long package description – this file is what contains the latter: A detailed description that should give you a good idea whether this port would satisfy your needs:

% cat /usr/ports/archivers/zstd/pkg-descr
Zstd, short for Zstandard, is a real-time compression algorithm providing
high compression ratios.  It offers a very wide range of compression vs.
speed trade-offs while being backed by a very fast decoder.  It offers

Then there’s a file called distinfo. It lists all files that need to be downloaded to build the port (usually the program’s source code). It also contains a checksum and the file’s size to make sure that the valid file is being used (an archive could get corrupted during the transfer or you could even get an archive that somebody tempered with!):

% cat /usr/ports/archivers/zstd/distinfo 
TIMESTAMP = 1503324578
SHA256 (facebook-zstd-v1.3.1_GH0.tar.gz) = 312fb9dc75668addbc9c8f33c7fa198b0fc965c576386b8451397e06256eadc6
SIZE (facebook-zstd-v1.3.1_GH0.tar.gz) = 1513767

There’s usually also pkg-plist. It lists all the files that are installed by the port:

% cat /usr/ports/archivers/zstd/pkg-plist 

And finally there’s the Makefile. This is where all the magic happens. If you’re a programmer or you have built software from source before, there’s a high chance that you’re at least somewhat familiar with a tool called make. It processes Makefiles and then does as told by those. While it’s most often used to compile software it can actually be used for a wide variety of tasks.

If you don’t have at least some experience with them, Makefiles look pretty much obscure and creating them seems like a black art. If you’ve ever looked at a complicated Makefile, you may be worried to hear that to use ports you have to use make. Don’t be. The people who take care of the Ports infrastructure are the ones who really need to know how to deal with all the nuts and bolts of make. They’ve already solved all the common tasks so that the porters (those people who create the actual ports) can rely on it. This is done by including other Makefiles and it manages to hide away all the scariness. And for you as a user things are even simpler as you can just use what others created for you!

Let’s take a look at the Makefile for our example port:

% cat /usr/ports/archivers/zstd/Makefile 
# Created by: John Marino <>
# $FreeBSD: head/archivers/zstd/Makefile 448492 2017-08-21 20:44:02Z sunpoet $

CATEGORIES=	archivers

COMMENT=	Zstandard - Fast real-time compression algorithm

	@${REINPLACE_CMD} -e 's|INSTALL_|BSD_&|' ${WRKSRC}/lib/Makefile ${WRKSRC}/programs/Makefile

.include <>

Now that doesn’t look half bad for a Makefile, does it? In fact it’s mostly just defining Variables! The only line that looks somewhat complex is the “post-patch” command (which is also less terrifying than it first looks – if you know sed you can surely guess what it’ll do).

There can actually be more files in some ports. If FreeBSD-specific patches are required to build the port, those are included in the ports tree. You can find them in a sub-directory called files located in the port’s directory. Here’s an example:

% ls /usr/ports/editors/vim/files/
patch-src-auto-configure        vietnamese_viscii.vim          vimrc

The patches there are named after the files that they apply to. Every patch in the files directory is automatically applied when building the port.

What’s next?

Alright. With that we’ve got a basic overview of what Ports are covered. The next post will show how to actually use them to build and install software.