FreeBSD: Building software from ports (1/2)

In my previous two (link) posts (link) I wrote about using Pkg, FreeBSD’s package manager.

Pre-built binary packages are convenient to use but sometimes you need some more flexibility, want an application that cannot not be distributed in binary form due to license issues (or have some other requirements). Building software by hand is certainly possible – but with all the things involved, this can be a rather tedious process. It’s also slow, error-prone and there’s often no clean way to get rid of that stuff again. FreeBSD Ports to the rescue!

This first part is meant as a soft introduction to FreeBSD’s ports, assuming no prior knowledge (if I fail to explain something, feel free to comment on this post). It will give you enough background information to understand ports enough to start using them in the next article.

What “Ports” are

When programmers talk about porting something over, what they originally meant is this: Take an application that was written with one processor architecture in mind (say i386) and modify the source so that it runs on another (arm64 for example) afterwards. The term “porting” is also used when modifying the source of any program to make it run on another OS. The version that runs on the other architecture/OS is called a port of the original program to a different platform.

FreeBSD uses the term slightly differently. There’s a lot of software written e.g. for Linux that will build and work on FreeBSD just fine as it is. Even though it does not require any changes, that software might be ported to FreeBSD. So in this case “porting” does not mean “make it work at all” but make it easily available. This is done by creating a port for any program. That term doesn’t mean a variant of the source code in this case but rather a means to give you easy access to that software on FreeBSD.

So what is a port in FreeBSD? Actually a port is a directory with a bunch of files in it. The heart of it is one file that basically is a recipe if you will. That recipe contains everything needed to build and install the port (and thus have the application installed on your machine in the end). Following this metaphor you could think of all the ports as a big cookbook. Formally it is known as the Ports collection. All those files in your filesystem related to ports are refered to as the Ports tree.

How to get the Ports tree

There are several options to obtain a copy of the ports tree. When you install FreeBSD you can decide whether or not to install it, too. I usually don’t do that because on systems that use binary packages only. It wastes only about 300 MB of space, but more importantly consists of almost 170.000 files (watch your inodes on embedded devices!). Take a look at /usr/ports: If that directory is empty your system is currently missing the ports tree.

The simplest way to get it is by using portsnap:

# portsnap fetch extract

If you want to update the tree later, you can use:

# portsnap fetch update

Another way is to use Subversion. This is more flexible: With portsnap you always get the current tree while Subversion also allows you to checkout older revisions, too. If you plan to become a ports developer, you will probably want to use Subversion for tools like svndiff. If you just want to use ports, portsnap should actually suffice. All currently supported versions of FreeBSD contain a light-weight version of Subversion called svnlite.

Here’s how to checkout the latest tree:

# svnlite checkout https://svn.freebsd.org/ports/head /usr/ports

If you want to update it later run:

# svnlite update /usr/ports

Old versions of the tree

You normally shouldn’t need these but it’s good to know that they exist. Using Subversion you can also retrieve old trees. Be sure that /usr/ports is empty (including for Subversion’s dot directories) or Subversion will see that there’s already something there and won’t do the checkout. If for example you want the ports tree as it existed in 2016Q4, you can retrieve it like this:

# svnlite checkout https://svn.freebsd.org/ports/branches/2016Q4 /usr/ports

There are also several tags available that allow to get certain trees. Maybe you want to see which ports were available when FreeBSD 9.2 was released. Get the tree like this:

# svnlite checkout https://svn.freebsd.org/ports/tags/RELEASE_9_2_0 /usr/ports

And if you need the last tree that is guaranteed to work with 9.x there’s another special tag for it:

# svnlite checkout https://svn.freebsd.org/ports/tags/RELEASE_9_EOL /usr/ports

Keep in mind though that using old trees is risky because they contain program versions with vulnerabilities that have since been found! Also mind that it’s NOT a smart thing to simply get the tree for RELEASE_7_EOL because it still holds a port for PHP 5.2 and you thought that it would be cool to offer your customers as many versions as possible. Yes, it may be possible that you can still build it if you invest some manual work. But no, that doesn’t make it a good idea at all.

Oh, and don’t assume that old ports trees will be of any use on modern versions of FreeBSD! The ports architecture changed quite a bit over time, the most notable change being the replacement of the old pkg_* tools with the new Pkg. Ports older than a certain time definitely won’t build in their old, unmodified state today (and I say it again: You really shouldn’t bother unless you have a very special case).

Port organization

Take a look at the contents of /usr/ports on a system that has the tree installed. You will find over 60 directories there. There are a few special ones like distfiles (where tarballs with program’s source code get stored – might be missing initially) or Mk that holds include files for the ports infrastructure. The others are categories.

If you’re looking for a port for Firefox, that will be in www. GIMP is in graphics and it’s probably no surprise that Audacious (a music player) can be found in audio. Some program’s categories will be less obvious. LibreOffice is in editors which is not so bad. But help2man for example is in misc and not in converters or devel as at least I would expect if I didn’t know. In general however after a while of working with ports you will have a pretty good chance to guess where things are.

Say we are interested in the port for the window manager Sawfish for example. It’s located in /usr/ports/x11-wm/sawfish. Let’s take a closer look at that location and take it apart:

/usr/ports is the “ports directory”.
x11-wm (short for X11 window managers) is the category.
sawfish is the individual port’s name.

When referring to where a port lives, you can omit the ports directory since everybody is assumed to know where it is. The important information when identifying a port is the category and the name. Together those form what is known as the port origin (x11-wm/sawfish in our case).

How to find a port in the tree

There are multiple methods to find out the origin for the port you are looking for. Probably the simplest one is using whereis. If we didn’t know that sawfish is in x11-wm/sawfish we could do this:

% whereis sawfish
sawfish: /usr/ports/x11-wm/sawfish

This does however only work if you know the exact name of the port. And there’s a little more to it: Sometimes the name of a port and a package differ! This is often the case for Python-based packages. I have SaltStack installed, for example. It’s a package called py27-salt:

% pkg info -x salt
py27-salt-2017.7.1_1

If we were to look for that, we wouldn’t find it:

% whereis py27-salt
py27-salt:

So where is the port for the package?

% pkg info py27-salt
py27-salt-2017.7.1_1
Name           : py27-salt
[...]
Origin         : sysutils/py-salt
[...]

Here you can see that the port’s name is py-salt! The “27” gets added when the package is created and reflects the version of Python that it’s build against. You may also see some py3-xyz ports. In those cases the name reflects that the port cannot be built with Python 2.x. The package will still be called py36-xyz, though (or whatever the default Python 3.x version is at that time)!

When discussing package management I recommended FreshPorts and when working with ports it can be useful, too. Search for some program’s name and it might be easier for you to find the package name and the port origin for it!

What a port looks like

Let’s take a look at the port for the zstd compression utility:

% ls /usr/ports/archivers/zstd/
distinfo	Makefile	pkg-descr	pkg-plist

So what have we here? The simplest file is pkg-descr. Each package has a short and a long package description – this file is what contains the latter: A detailed description that should give you a good idea whether this port would satisfy your needs:

% cat /usr/ports/archivers/zstd/pkg-descr
Zstd, short for Zstandard, is a real-time compression algorithm providing
high compression ratios.  It offers a very wide range of compression vs.
speed trade-offs while being backed by a very fast decoder.  It offers
[...]

Then there’s a file called distinfo. It lists all files that need to be downloaded to build the port (usually the program’s source code). It also contains a checksum and the file’s size to make sure that the valid file is being used (an archive could get corrupted during the transfer or you could even get an archive that somebody tempered with!):

% cat /usr/ports/archivers/zstd/distinfo 
TIMESTAMP = 1503324578
SHA256 (facebook-zstd-v1.3.1_GH0.tar.gz) = 312fb9dc75668addbc9c8f33c7fa198b0fc965c576386b8451397e06256eadc6
SIZE (facebook-zstd-v1.3.1_GH0.tar.gz) = 1513767

There’s usually also pkg-plist. It lists all the files that are installed by the port:

% cat /usr/ports/archivers/zstd/pkg-plist 
bin/unzstd
bin/zstd
bin/zstdcat
[...]
lib/libzstd.so.%%PORTVERSION%%
libdata/pkgconfig/libzstd.pc
man/man1/unzstd.1.gz
man/man1/zstd.1.gz
man/man1/zstdcat.1.gz

And finally there’s the Makefile. This is where all the magic happens. If you’re a programmer or you have built software from source before, there’s a high chance that you’re at least somewhat familiar with a tool called make. It processes Makefiles and then does as told by those. While it’s most often used to compile software it can actually be used for a wide variety of tasks.

If you don’t have at least some experience with them, Makefiles look pretty much obscure and creating them seems like a black art. If you’ve ever looked at a complicated Makefile, you may be worried to hear that to use ports you have to use make. Don’t be. The people who take care of the Ports infrastructure are the ones who really need to know how to deal with all the nuts and bolts of make. They’ve already solved all the common tasks so that the porters (those people who create the actual ports) can rely on it. This is done by including other Makefiles and it manages to hide away all the scariness. And for you as a user things are even simpler as you can just use what others created for you!

Let’s take a look at the Makefile for our example port:

% cat /usr/ports/archivers/zstd/Makefile 
# Created by: John Marino <marino@FreeBSD.org>
# $FreeBSD: head/archivers/zstd/Makefile 448492 2017-08-21 20:44:02Z sunpoet $

PORTNAME=	zstd
PORTVERSION=	1.3.1
DISTVERSIONPREFIX=	v
CATEGORIES=	archivers

MAINTAINER=	sunpoet@FreeBSD.org
COMMENT=	Zstandard - Fast real-time compression algorithm

LICENSE=	BSD3CLAUSE GPLv2
[...]
post-patch:
	@${REINPLACE_CMD} -e 's|INSTALL_|BSD_&|' ${WRKSRC}/lib/Makefile ${WRKSRC}/programs/Makefile

.include <bsd.port.mk>

Now that doesn’t look half bad for a Makefile, does it? In fact it’s mostly just defining Variables! The only line that looks somewhat complex is the “post-patch” command (which is also less terrifying than it first looks – if you know sed you can surely guess what it’ll do).

There can actually be more files in some ports. If FreeBSD-specific patches are required to build the port, those are included in the ports tree. You can find them in a sub-directory called files located in the port’s directory. Here’s an example:

% ls /usr/ports/editors/vim/files/
patch-src-auto-configure        vietnamese_viscii.vim
patch-src-installml.sh          vimrc

The patches there are named after the files that they apply to. Every patch in the files directory is automatically applied when building the port.

What’s next?

Alright. With that we’ve got a basic overview of what Ports are covered. The next post will show how to actually use them to build and install software.

Advertisements

FreeBSD package management with Pkg (2/2)

The previous article covered basic operations with FreeBSD’s Pkg tool. This second part will deal with some more advanced (or rather intermediate, actually) functionality.

Good code travels well

My previous two articles have been linked to from the DragonFly Digest (a very valuable resource for topics in BSD and the IT in general that I’ve been reading for years now and would like to say “thanks!”) again. Justin Sherrill pointed out that everything applies to DragonFlyBSD as well – they have adopted Pkg quite a while ago. And in fact you benefit from knowing your way around with Pkg in a lot of places:

FreeBSD obviously and a lot of FreeBSD-derived operating systems like OPNsense and HardenedBSD as well as desktop-oriented offspring like GhostBSD and TrueOS.

But as mentioned before, DragonFlyBSD uses it, too. And thanks to the new (and extremely exciting, IMO!) Ravenports project it has already come to Linux and will be available on even more platforms in the future! So getting familiar with it is certainly not a waste of time.

Package versioning

Before we start updating packages, let’s take a look at the versioning scheme. The way FreeBSD versions its packages can be a bit confusing if you first see it. Here’s a sample package with a rather complicated version string:

# pkg search opensmtpd | grep OpenBSD
opensmtpd-5.9.2p1_3,1          Security- and simplicity-focused SMTP server from OpenBSD

opensmtpd-5.9.2p1_3,1 – what does that all mean? Well, first we have the package name: opensmtpd, followed by a minus. Then there’s the upstream version of the program, 5.9.2p1 in this case.

Then there’s the underscore and another number: _3 in this case. This indicates our package is “revision 3”. Any new package starts with a revision of 0. If a port is revised (probably to correct a mistake, add more configure options, etc), the revision number is bumped. So this port has been revised three times without changing the actual upstream version.

And finally, separated by a comma, we have what is called the “epoch”. It is used in such cases where the upstream versioning changes. Any package with an epoch of 1 is considered newer than a package without any epoch. Even higher epoch numbers are considered even newer but this is rare. When do you need this? Let’s assume some project released a version of 7.2017 but decided that it would be a good idea to release the next version as 5.0. For Pkg it looks like the first one is newer (as it has a higher version number). In such a case you’d set an epoch to make Pkg understand that in fact the other one is the more up-to-date package.

Updating packages

I covered updating the repository information before. Update the actual packages with pkg upgrade:

# pkg upgrade
Updating Synth repository catalogue...
Synth repository is up to date.
All repositories are up to date.
Checking for upgrades (30 candidates): 100%
Processing candidates (30 candidates): 100%
Checking integrity... done (0 conflicting)
The following 30 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        xinit: 1.3.4,1 -> 1.3.4_1,1
        xerces-c3: 3.1.4 -> 3.2.0_2
        virtualbox-ose: 5.1.26 -> 5.1.26_1
        vim: 8.0.0962 -> 8.0.1035
        sudo: 1.8.20p2_3 -> 1.8.21p1
        sqlite3: 3.20.0_2 -> 3.20.1
        rubygem-net-ssh: 4.1.0,2 -> 4.2.0,2
        rubygem-multi_json: 1.12.1 -> 1.12.2
        ruby23-gems: 2.6.12 -> 2.6.13
        pulseaudio: 10.0_4 -> 11.0
        pciids: 20170727 -> 20170825
        p11-kit: 0.23.7 -> 0.23.8
        open-vm-tools: 10.1.5_1,2 -> 10.1.10,2
        nano: 2.8.6 -> 2.8.7
        mesa-libs: 17.1.7 -> 17.1.8
        mesa-dri: 17.1.7 -> 17.1.8
        libreoffice: 5.3.5_1 -> 5.3.6
        libidn2: 2.0.3 -> 2.0.4
        libgcrypt: 1.8.0 -> 1.8.1
        libdrm: 2.4.82,1 -> 2.4.83,1
        hunspell: 1.6.1_1 -> 1.6.2
        harfbuzz-icu: 1.4.8 -> 1.5.1
        harfbuzz: 1.4.8 -> 1.5.1
        gdk-pixbuf2: 2.36.6 -> 2.36.9
        e2fsprogs: 1.43.5 -> 1.43.5_1
        doas: 6.0p0 -> 6.0p1
        chromium: 60.0.3112.101 -> 60.0.3112.113
        atril: 1.18.0_1 -> 1.18.1

Installed packages to be REINSTALLED:
        keybinder-0.3.1 (options changed)
        apache-xml-security-c-1.7.3 (needed shared library changed)

Number of packages to be upgraded: 28
Number of packages to be reinstalled: 2

The process will require 2 MiB more space.

Proceed with this action? [y/N]:

The packages to be UPGRADED section is pretty obvious: There’s a newer version available. But there are also two packages in this example that are being reinstalled even though no new version is available. Pkg gives the reason for this in parentheses:

Keybinder will be reinstalled because it was compiled with other compile-time options than before (more about this in the next post). The second one depends on xerces-c3, a package in the list of upgradable packages, which is why apache-xml-security-c was rebuilt against the new version of the library.

There are other reasons that packages are to be reinstalled; if you upgraded your OS from one major version to another, the reason might be “ABI has changed”. It’s also possible that some packages will be deinstalled for an upgrade. This is usually because they conflict with another package that is to be installed. This also means: Do look at what the update is going to do! There is the chance that it would do something that you didn’t intend to.

Will this update cause me trouble?

You can never know for sure. But there is a means to learn about known issues beforehand. For your important applications it is a good idea to read the so-called “UPDATING information”. This is a short text (or some of them) which might contain a heads-up that can be critical to know. To view it, use pkg updating. Here’s an old example showing how bad it could be to have missed it:

# pkg updating apache22                 
20140713:                                          
  AFFECTS: users of www/apache22                   
  AUTHOR: ohauer@FreeBSD.org                       
                                                   
  The default version was changed from www/apache22 to www/apache24,                                   
  pre-build apache modules and web applications will also reflect this!                                
                                                   
  In case ports are build by yourself and apache22 is required                                         
  use the following command to keep apache22 as default.                                               
                                                   
  # echo "DEFAULT_VERSIONS+=apache=2.2" >> /etc/make.conf

Having missed that one would have had very bad effects… For such reasons it’s good practice to read the UPDATING info. You don’t actually have to read it and will probably get away with it for quite some time. But it’s there for your benefit. So if you choose to ignore it, don’t complain if an update finally bites when it finds you off guard!

Blocking updates

Let’s stick to the previous example and say that we want to do the update – but LibreOffice should not be touched because we’re working on an important document currently and don’t want to risk layout breakage (minor updates should be no problem but bigger updates are known to sometimes cause trouble). What to do in that case?

Let’s lock the package using pkg lock:

# pkg lock libreoffice
libreoffice-5.3.5_1: lock this package? [y/N]: y
Locking libreoffice-5.3.5_1

Attempting the upgrade again, Pkg should now show only 27 candidates and leave LibreOffice alone. There are a few good reasons to lock a package – and a lot of bad ones. Resort to locking packages when necessary but don’t trifle with it because you’re effectively cutting yourself off from updates on some packages. Those could have dependencies. Probably dependencies that they share with other packages. You can see how this gets a lot bigger than “just that one package” rather quickly.

Also if you decide to use locking, make sure to look for locked packages now and then and think over if the lock is still needed! If not, release the lock. But how to find out which packages are locked? Pkg info can help us out:

# pkg info -k -a | grep yes             
libreoffice-5.3.5_1            yes

Unlocking works just like you’d probably expect it to:

# pkg unlock libreoffice                                                                                                                                                                           
libreoffice-5.3.5_1: unlock this package? [y/N]: y
Unlocking libreoffice-5.3.5_1

Package comments

We’ve locked LibreOffice above – but how do we remember in four months or so why it was locked? This is what we can use an annotation for. Set one with pkg annotate:

# pkg annotate -A libreoffice locked-pkgs "This package was locked on 09/10 until I finally finish the manuscript for my fantasy novel!"                                                           
libreoffice-5.3.5_1: Add annotation tagged: locked-pkgs with value: This package was locked on 09/10 until I finally finish the manuscript for my fantasy novel!? [y/N]: y
libreoffice-5.3.5_1: added annotation tagged: locked-pkgs

The argument “-A” is to set an annotation to the following package. “locked-pkgs” is a tag – you could call it whatever you want. And finally the last field is the actual comment string.

Using pkg info and the package name will display the comment among a lot of other information. But it might make more sense to look for all packages that have an annotation with a certain tag:

# pkg annotate -a -S locked-pkgs
libreoffice-5.3.5_1: Tag: locked-pkgs Value: This package was locked on 09/10 until I finally finish the manuscript for my fantasy novel!

If you no longer need the annotation, delete it like this:

# pkg annotate -D libreoffice locked-pkgs                                                                                                                                                          
libreoffice-5.3.5_1: Delete annotation tagged: locked-pkgs? [y/N]: y
libreoffice-5.3.5_1: Deleted annotation tagged: locked-pkgs

Are those updates important?

Some updates mean new features, others mean fixing of critical security holes. How are you supposed to know? The easy way is to ask Pkg! Use pkg audit and it will tell you about known vulnerabilities of the software installed on your system:

pkg audit
libgcrypt-1.8.0 is vulnerable:
libgcrypt -- side-channel attack vulnerability
CVE: CVE-2017-0379
WWW: https://vuxml.FreeBSD.org/freebsd/22f28bb3-8d98-11e7-8c37-e8e0b747a45a.html

chromium-60.0.3112.101 is vulnerable:
chromium -- multiple vulnerabilities
CVE: CVE-2017-5120
CVE: CVE-2017-5119
CVE: CVE-2017-5118
CVE: CVE-2017-5117
CVE: CVE-2017-5116
CVE: CVE-2017-5115
CVE: CVE-2017-5114
CVE: CVE-2017-5113
CVE: CVE-2017-5112
CVE: CVE-2017-5111
WWW: https://vuxml.FreeBSD.org/freebsd/e1100e63-92f7-11e7-bd95-e8e0b747a45a.html
[...]

No way back?

Are you not feeling completely confident about an update? Does your customer demand “a way back” in case something goes wrong? You can use pkg create to package already installed software:

# pkg create chromium-60.0.3112.101
Creating package for chromium-60.0.3112.101

In this example I’ve packaged Chromium before updating so that I could reinstall the old version. Keep in mind, though, that this is just an example. If dependencies changed as well, you might not be able to use the old version, even when you reinstalled it! If you want to be really, really cautious, you can use pkg create -a to create packages of all the software currently installed on your system!

The package(s) is/are created in the current directory. I just deinstalled Chromium after creating the package and now want it back. To install software directly from a package (and not a repo), use pkg add:

# pkg add chromium-60.0.3112.101.txz
Installing chromium-60.0.3112.101...
Extracting chromium-60.0.3112.101: 100%
Message from chromium-60.0.3112.101:
For correct operation, shared memory support has to be enabled
in Chromium by performing the following command as root :

sysctl kern.ipc.shm_allow_removed=1

To preserve this setting across reboots, append the following
to /etc/sysctl.conf :

kern.ipc.shm_allow_removed=1

Finding the package a file belongs to

In many cases you can probably tell from the path and name of a file which package it belongs to. But sometimes you may wonder: Where does this come from? This is where pkg which is really helpful. Let’s pick a file with a non-obvious name and pretend we don’t know what it is. We better ask pkg where it belongs:

% pkg which /usr/local/etc/drirc     
/usr/local/etc/drirc was installed by package mesa-dri-17.1.7

Ah, mesa! We better leave that one alone.

Repositories

If you looked closely at the output of my upgrade command, you have seen mention of a repo called Synth. I’ll cover that in a later post. But there is something you might want to know about the ordinary repos, too. Modern FreeBSD provides two package repositories: quarterly and latest. The later always holds the newest packages, the former gets version updates every three months and only security fixes in-between. The quarterly repository is a good choice for people who don’t want the newest software at all times but prefer a slower-moving environment. Since version 10.2 quarterly is the default.

If you want to use the packages from latest, you have to configure pkg to use it. Take a look at the file /etc/pkg/FreeBSD.conf to get an idea of how repo configuration looks like. Then create the necessary directory and another configuration file to overwrite the default:

# mkdir -p /usr/local/etc/pkg/repos
# vi /usr/local/etc/pkg/repos/FreeBSD.conf

Put the following lines in that file:

FreeBSD: {
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest"
}

Now use pkg update to refresh the repository database.

What’s next?

There’s a lot more that Pkg can do – and we haven’t even touched its main configuration! But the two posts were just meant to introduce you to FreeBSD package management (and chances are that you already know more now than many admins who occasionally use FreeBSD). I might or might not write about more features of Pkg in the future. But next stop now: The ports tree.

Edit: I got the patch level wrong with the version schema as leper pointed out in the comments. The wrong claim was removed.