In my previous two (link) posts (link) I wrote about using Pkg, FreeBSD’s package manager.
Pre-built binary packages are convenient to use but sometimes you need some more flexibility, want an application that cannot not be distributed in binary form due to license issues (or have some other requirements). Building software by hand is certainly possible – but with all the things involved, this can be a rather tedious process. It’s also slow, error-prone and there’s often no clean way to get rid of that stuff again. FreeBSD Ports to the rescue!
This first part is meant as a soft introduction to FreeBSD’s ports, assuming no prior knowledge (if I fail to explain something, feel free to comment on this post). It will give you enough background information to understand ports enough to start using them in the next article.
What “Ports” are
When programmers talk about porting something over, what they originally meant is this: Take an application that was written with one processor architecture in mind (say i386) and modify the source so that it runs on another (arm64 for example) afterwards. The term “porting” is also used when modifying the source of any program to make it run on another OS. The version that runs on the other architecture/OS is called a port of the original program to a different platform.
FreeBSD uses the term slightly differently. There’s a lot of software written e.g. for Linux that will build and work on FreeBSD just fine as it is. Even though it does not require any changes, that software might be ported to FreeBSD. So in this case “porting” does not mean “make it work at all” but make it easily available. This is done by creating a port for any program. That term doesn’t mean a variant of the source code in this case but rather a means to give you easy access to that software on FreeBSD.
So what is a port in FreeBSD? Actually a port is a directory with a bunch of files in it. The heart of it is one file that basically is a recipe if you will. That recipe contains everything needed to build and install the port (and thus have the application installed on your machine in the end). Following this metaphor you could think of all the ports as a big cookbook. Formally it is known as the Ports collection. All those files in your filesystem related to ports are refered to as the Ports tree.
How to get the Ports tree
There are several options to obtain a copy of the ports tree. When you install FreeBSD you can decide whether or not to install it, too. I usually don’t do that because on systems that use binary packages only. It wastes only about 300 MB of space, but more importantly consists of almost 170.000 files (watch your inodes on embedded devices!). Take a look at /usr/ports: If that directory is empty your system is currently missing the ports tree.
The simplest way to get it is by using portsnap:
# portsnap fetch extract
If you want to update the tree later, you can use:
# portsnap fetch update
Another way is to use Subversion. This is more flexible: With portsnap you always get the current tree while Subversion also allows you to checkout older revisions, too. If you plan to become a ports developer, you will probably want to use Subversion for tools like svndiff. If you just want to use ports, portsnap should actually suffice. All currently supported versions of FreeBSD contain a light-weight version of Subversion called svnlite.
Here’s how to checkout the latest tree:
# svnlite checkout https://svn.freebsd.org/ports/head /usr/ports
If you want to update it later run:
# svnlite update /usr/ports
Old versions of the tree
You normally shouldn’t need these but it’s good to know that they exist. Using Subversion you can also retrieve old trees. Be sure that /usr/ports is empty (including for Subversion’s dot directories) or Subversion will see that there’s already something there and won’t do the checkout. If for example you want the ports tree as it existed in 2016Q4, you can retrieve it like this:
# svnlite checkout https://svn.freebsd.org/ports/branches/2016Q4 /usr/ports
There are also several tags available that allow to get certain trees. Maybe you want to see which ports were available when FreeBSD 9.2 was released. Get the tree like this:
# svnlite checkout https://svn.freebsd.org/ports/tags/RELEASE_9_2_0 /usr/ports
And if you need the last tree that is guaranteed to work with 9.x there’s another special tag for it:
# svnlite checkout https://svn.freebsd.org/ports/tags/RELEASE_9_EOL /usr/ports
Keep in mind though that using old trees is risky because they contain program versions with vulnerabilities that have since been found! Also mind that it’s NOT a smart thing to simply get the tree for RELEASE_7_EOL because it still holds a port for PHP 5.2 and you thought that it would be cool to offer your customers as many versions as possible. Yes, it may be possible that you can still build it if you invest some manual work. But no, that doesn’t make it a good idea at all.
Oh, and don’t assume that old ports trees will be of any use on modern versions of FreeBSD! The ports architecture changed quite a bit over time, the most notable change being the replacement of the old pkg_* tools with the new Pkg. Ports older than a certain time definitely won’t build in their old, unmodified state today (and I say it again: You really shouldn’t bother unless you have a very special case).
Port organization
Take a look at the contents of /usr/ports on a system that has the tree installed. You will find over 60 directories there. There are a few special ones like distfiles (where tarballs with program’s source code get stored – might be missing initially) or Mk that holds include files for the ports infrastructure. The others are categories.
If you’re looking for a port for Firefox, that will be in www. GIMP is in graphics and it’s probably no surprise that Audacious (a music player) can be found in audio. Some program’s categories will be less obvious. LibreOffice is in editors which is not so bad. But help2man for example is in misc and not in converters or devel as at least I would expect if I didn’t know. In general however after a while of working with ports you will have a pretty good chance to guess where things are.
Say we are interested in the port for the window manager Sawfish for example. It’s located in /usr/ports/x11-wm/sawfish. Let’s take a closer look at that location and take it apart:
/usr/ports is the “ports directory”.
x11-wm (short for X11 window managers) is the category.
sawfish is the individual port’s name.
When referring to where a port lives, you can omit the ports directory since everybody is assumed to know where it is. The important information when identifying a port is the category and the name. Together those form what is known as the port origin (x11-wm/sawfish in our case).
How to find a port in the tree
There are multiple methods to find out the origin for the port you are looking for. Probably the simplest one is using whereis. If we didn’t know that sawfish is in x11-wm/sawfish we could do this:
% whereis sawfish sawfish: /usr/ports/x11-wm/sawfish
This does however only work if you know the exact name of the port. And there’s a little more to it: Sometimes the name of a port and a package differ! This is often the case for Python-based packages. I have SaltStack installed, for example. It’s a package called py27-salt:
% pkg info -x salt py27-salt-2017.7.1_1
If we were to look for that, we wouldn’t find it:
% whereis py27-salt py27-salt:
So where is the port for the package?
% pkg info py27-salt py27-salt-2017.7.1_1 Name : py27-salt [...] Origin : sysutils/py-salt [...]
Here you can see that the port’s name is py-salt! The “27” gets added when the package is created and reflects the version of Python that it’s build against. You may also see some py3-xyz ports. In those cases the name reflects that the port cannot be built with Python 2.x. The package will still be called py36-xyz, though (or whatever the default Python 3.x version is at that time)!
When discussing package management I recommended FreshPorts and when working with ports it can be useful, too. Search for some program’s name and it might be easier for you to find the package name and the port origin for it!
What a port looks like
Let’s take a look at the port for the zstd compression utility:
% ls /usr/ports/archivers/zstd/ distinfo Makefile pkg-descr pkg-plist
So what have we here? The simplest file is pkg-descr. Each package has a short and a long package description – this file is what contains the latter: A detailed description that should give you a good idea whether this port would satisfy your needs:
% cat /usr/ports/archivers/zstd/pkg-descr Zstd, short for Zstandard, is a real-time compression algorithm providing high compression ratios. It offers a very wide range of compression vs. speed trade-offs while being backed by a very fast decoder. It offers [...]
Then there’s a file called distinfo. It lists all files that need to be downloaded to build the port (usually the program’s source code). It also contains a checksum and the file’s size to make sure that the valid file is being used (an archive could get corrupted during the transfer or you could even get an archive that somebody tempered with!):
% cat /usr/ports/archivers/zstd/distinfo TIMESTAMP = 1503324578 SHA256 (facebook-zstd-v1.3.1_GH0.tar.gz) = 312fb9dc75668addbc9c8f33c7fa198b0fc965c576386b8451397e06256eadc6 SIZE (facebook-zstd-v1.3.1_GH0.tar.gz) = 1513767
There’s usually also pkg-plist. It lists all the files that are installed by the port:
% cat /usr/ports/archivers/zstd/pkg-plist bin/unzstd bin/zstd bin/zstdcat [...] lib/libzstd.so.%%PORTVERSION%% libdata/pkgconfig/libzstd.pc man/man1/unzstd.1.gz man/man1/zstd.1.gz man/man1/zstdcat.1.gz
And finally there’s the Makefile. This is where all the magic happens. If you’re a programmer or you have built software from source before, there’s a high chance that you’re at least somewhat familiar with a tool called make. It processes Makefiles and then does as told by those. While it’s most often used to compile software it can actually be used for a wide variety of tasks.
If you don’t have at least some experience with them, Makefiles look pretty much obscure and creating them seems like a black art. If you’ve ever looked at a complicated Makefile, you may be worried to hear that to use ports you have to use make. Don’t be. The people who take care of the Ports infrastructure are the ones who really need to know how to deal with all the nuts and bolts of make. They’ve already solved all the common tasks so that the porters (those people who create the actual ports) can rely on it. This is done by including other Makefiles and it manages to hide away all the scariness. And for you as a user things are even simpler as you can just use what others created for you!
Let’s take a look at the Makefile for our example port:
% cat /usr/ports/archivers/zstd/Makefile
# Created by: John Marino <marino@FreeBSD.org>
# $FreeBSD: head/archivers/zstd/Makefile 448492 2017-08-21 20:44:02Z sunpoet $
PORTNAME= zstd
PORTVERSION= 1.3.1
DISTVERSIONPREFIX= v
CATEGORIES= archivers
MAINTAINER= sunpoet@FreeBSD.org
COMMENT= Zstandard - Fast real-time compression algorithm
LICENSE= BSD3CLAUSE GPLv2
[...]
post-patch:
@${REINPLACE_CMD} -e 's|INSTALL_|BSD_&|' ${WRKSRC}/lib/Makefile ${WRKSRC}/programs/Makefile
.include <bsd.port.mk>
Now that doesn’t look half bad for a Makefile, does it? In fact it’s mostly just defining Variables! The only line that looks somewhat complex is the “post-patch” command (which is also less terrifying than it first looks – if you know sed you can surely guess what it’ll do).
There can actually be more files in some ports. If FreeBSD-specific patches are required to build the port, those are included in the ports tree. You can find them in a sub-directory called files located in the port’s directory. Here’s an example:
% ls /usr/ports/editors/vim/files/ patch-src-auto-configure vietnamese_viscii.vim patch-src-installml.sh vimrc
The patches there are named after the files that they apply to. Every patch in the files directory is automatically applied when building the port.
What’s next?
Alright. With that we’ve got a basic overview of what Ports are covered. The next post will show how to actually use them to build and install software.
