FreeBSD package building pt. 5: Sophisticated Synth

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/freebsd_package_building_pt5.gmi

In the previous posts of this series, there was an introduction to package building on FreeBSD and we discussed basic Synth usage. The program’s configuration, working with compiler cache and using the tool to update the installed applications was covered as well. We also discussed Synth web reports, serving repositories over HTTP and building package sets. We also took a brief look at the logs to find out why some ports failed.

In this article we are going to sign our package repository and explore make.conf options. We will also add an additional profile to build packages for an obsolete FreeBSD release that is 32-bit and automate package building with cron.

Changing the structure

So far we’ve only considered building basically one local package set (even though we’ve shared it). If we want to have a single build server manage multiple package sets, we will need a somewhat more complex directory structure to allow for separate repositories and such. Should you be building on a UFS system, it’s easy: Just create the additional directories. I’m using ZFS here, though, and need to think if I want to create the whole structure as several datasets or just two single datasets with custom mount? As usually there’s pros and cons for both. I’m going with the former here:

# rm -r /var/synth
# zfs create zroot/var/synth
# zfs create zroot/var/synth/www
# zfs create zroot/var/synth/www/log
# zfs create zroot/var/synth/www/log/13.0_amd64
# zfs create zroot/var/synth/www/packages
# zfs create zroot/var/synth/www/packages/13.0_amd64
# synth configure

Now we need to adapt the Synth configuration for the new paths:

B needs to be set to /var/synth/www/packages/13.0_amd64 and E to /var/synth/www/log/13.0_amd64. Normally I’d create a custom profile for that, but as I’m covering that a little later in this article, we’re going to abuse the LiveSystem default profile for now.

Next is re-configuring the webserver:

# vi /usr/local/etc/obhttpd.conf

Remove the block return directive in the location “/” block on the synth.local vhost and replace it with:

directory auto index

Then change the location to “/*”. I’m also removing the second location block. Create a new .htpasswd and bring over authentication to the main block if you want to.

# service obhttpd restart

Repository signing

To be able to use signing, we need a key pair available to Synth. Use the openssl command to create a private key, change permissions and then create a public key, too:

# openssl genrsa -out /usr/local/etc/synth/LiveSystem-private.key 2048
# chmod 0400 /usr/local/etc/synth/LiveSystem-private.key
# openssl rsa -pubout -in /usr/local/etc/synth/LiveSystem-private.key -out /usr/local/etc/synth/LiveSystem-public.key

Mind the filenames here! The LiveSystem part refers to the name of the profile we’re using. If you want to sign different repositories resulting from various profiles, make sure that you place the two key files for each of the profiles in /usr/local/etc/synth.

While you’re at it, consider either generating a self-signed TLS certificate or using Let’s Encrypt (if you have own a proper domain). If you opted to use TLS, change the webserver configuration once more to have it serve both the log and the package vhosts via HTTPS. There’s an example configuration (obhttpd.conf.sample) that comes with obhttpd in case you want to take a look. It covers HTTPS vhosts.

Alright! Since we changed the paths, we don’t currently have any repository to sign. Let’s build a popular browser now:

# synth build www/firefox

Firefox failed in the configure phase!

Firefox failed to build. This is what the log says:

DEBUG: Executing: `/usr/local/bin/cbindgen --version`
DEBUG: /usr/local/bin/cbindgen has version 0.18.0
ERROR: cbindgen version 0.18.0 is too old. At least version 0.19.0 is required.

Please update using 'cargo install cbindgen --force' or running
'./mach bootstrap', after removing the existing executable located at
/usr/local/bin/cbindgen.

===>  Script "configure" failed unexpectedly.
Please report the problem to gecko@FreeBSD.org [maintainer] and attach the
"/construction/xports/www/firefox/work/.build/config.log" including the output
of the failure of your make command. Also, it might be a good idea to provide
an overview of all packages installed on your system (e.g. a
/usr/local/sbin/pkg-static info -g -Ea).
*** Error code 1

Stop.
make: stopped in /xports/www/firefox



--------------------------------------------------
--  Termination
--------------------------------------------------
Finished: Friday, 11 JUN 2021 at 03:03:06 UTC
Duration: 00:03:06

Oh well! We’ve hit a problem in the ports tree. Somebody updated the Firefox port in our branch to a version that requires a newer cbindgen port than is available in the same branch! Breakage like this does happen sometimes (we’re all human after all). What to do about it? In our case: Ignore it as it’s only an example. Otherwise I’d advise you to update to a newer ports tree as these problems are usually quickly redeemed.

Synth is asking whether it should rebuild the repository. Yes, we want to do that. Then it asks if it should update the system with the newly built packages. And no, not now. Also note: The synth build command that we used here, is interactive and thus not well fit if you want to automate things:

Would you like to rebuild the local repository (Y/N)? y
Stand by, recursively scanning 1 port serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Would you like to upgrade your system with the new packages now (Y/N)? n

What else do we have to do to sign the repository? Nothing. Synth has already done that and even changed the local repository configuration to make the package manager verify the signature:

# tail -n 3 /usr/local/etc/pkg/00_synth.conf
  signature_type: PUBKEY,
  pubkey        : /usr/local/etc/synth/LiveSystem-public.key
}

That wasn’t so hard now, was it? You might want to know that Synth also supports using a signing server instead of signing locally. If this is something you’re interested in, do a man 1 synth and read the appropriate section.

Global options with make.conf

FreeBSD has two main configuration files that affect the compilation process when using the system compiler. The more general one, /etc/make.conf and another one that is only used when building FreeBSD from source: /etc/src.conf. Since we’re talking about building ports, we can ignore the latter.

There’s a manual page, make.conf(5), which describes some of the options that can be put into there. Most of the ones covered there are only relevant for building the system. Do by all means leave things like CFLAGS alone if you don’t know what you’re doing! Regarding the ports tree, it’s most useful to set or unset common options globally. It’s very tedious to set all the options for your ports manually like this:

# make -C /usr/ports/sysutils/tmux config-recursive

You need to do this for specific ports that you want to change the options for. But if there’s some that you have a global policy for, it’s better to use make.conf. Let’s say we want to never include documentation and examples in our ports. This would be done by adding the following line to /etc/make.conf:

OPTIONS_UNSET+=DOCS EXAMPLES

This affects all ports whether built by Synth or not as well as each and every Synth profile. Let’s say we also want no foreign language support in our packages for the default Synth profile (but in all others), we’d create the file /usr/local/etc/synth/LiveSystem-make.conf and put the following in there:

OPTIONS_UNSET+=NLS

That setting will build packages without NLS in addition to building without DOCS and EXAMPLES – if “LiveSystem” is the active profile.

If you want to build all ports that support it with e.g. the DEBUG option, add another line:

OPTION_SET+=DEBUG

Some common options that you might want to use include:

  • X11
  • CUPS
  • GTK3
  • QT5
  • JAVA
  • MANPAGES

After unsetting DOCS and EXAMPLES globally as well as NLS for the default profile, we’re going to rebuild the installed packages next:

# synth prepare-system

Package rebuild done

Note that Synth only rebuilt the packages that were affected by the changed options either directly or via their dependencies. For that reason only 219 of the 344 packages actually installed were rebuilt. If we now use pkg upgrade, this is what happens (see screenshot):

pkg upgrade will free space due to removing docs, examples and nls files

Ignore the 3 packages getting updated; this is due to packages that were skipped due to the Rust failure originally. That port has successfully built when we were trying to build Firefox, so our latest package runs built three more packages that have not been updated to, yet.

More interestingly: There’s 31 reinstalls. Most of them due to the package manager detecting changed options and one due to a change to a required shared library. It’s not hard to do the math and figure out that 31 is quite a bit less than 219. It’s a little less obvious that build-time dependencies count towards that greater number while they don’t appear in the count of packages that are eventually reinstalled. Still it’s true that Synth takes a “better safe than sorry” approach and tends to rebuild some packages that pkg(8) will end up not reinstalling. But this is not much of a problem, especially if you’re using ccache.

Alternative profiles

If you want to use Synth to build more than one specific set of packages for exactly one platform, you can. One way to achive this would be to always change Synth’s configuration. But that’s tedious and error prone. For that reason profiles exist. They allow you to have multiple different configurations available at the same time. If you’re simply running Synth from the command line like we’ve always done so far, it will use the configuration of the active profile.

To show off how powerful this is, we’re going to do something a little special: Building 32-bit packages for the no longer supported version of FreeBSD 12.1. Since amd64 CPUs are capable of running i386 programs, this does not even involve emulation. We need to create a couple of new directories first:

# mkdir /var/synth/www/packages/12.1_i386
# mkdir /var/synth/www/log/12.1_i386
# mkdir -p /var/synth/sysroot/12.1_i386

The last one is something that you might or might not be familiar with. A sysroot is a somewhat common term for – well, the root of a system. The sysroot of our running system is /. But we can put the data for other systems somewhere in our filesystem. If we put the base system of 32-bit 12.1-RELEASE into the directory created last, that’ll be a sysroot for 12.1 i386. Technically we don’t need all of the base system and could cherry-pick. It’s easier to simply use the whole thing, though:

# fetch -o /tmp/12.1-i386-base.txz http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/i386/12.1-RELEASE/base.txz
# tar -C /var/synth/sysroot/12.1_i386 -xf /tmp/12.1-i386-base.txz
# rm /tmp/12.1-i386-base.txz
# synth configure

Alright. Now we’re going to create an additional profile. To do so, press > (greater than key), then chose 2 (i.e. Create new profile) and give it a name like e.g. 12.1-i386. Then change the following three settings:

B: /var/synth/www/packages/12.1_i386
E: /var/synth/www/log/12.1_i386
G: /var/synth/sysroot/12.1_i386

That’s all, after you save the configuration you’re ready to go. Create a list of packages you want to build and let Synth do it’s thing:

# synth just-build /var/synth/pkglist.12.1_i386

The build will fail almost immediately. Why? Let’s take a look. Building pkg(8) failed and here’s why:

--------------------------------------------------------------------------------
--  Phase: check-sanity
--------------------------------------------------------------------------------
/!\ ERROR: /!\

Ports Collection support for your FreeBSD version has ended, and no ports are
guaranteed to build on this system. Please upgrade to a supported release.

No support will be provided if you silence this message by defining
ALLOW_UNSUPPORTED_SYSTEM.

*** Error code 1

Stop.
make: stopped in /xports/ports-mgmt/pkg

Ok, since we’re trying to build for a system that’s not supported anymore, the ports infrastructure warns us about that. We have to tell it to ignore that. How do we do that? You might have guessed: By using a make.conf for the profile we’re using for this set of packages:

# echo ALLOW_UNSUPPORTED_SYSTEM=1 > /usr/local/etc/synth/12.1-i386-make.conf

Then try again to build the set – and it will just work.

Successfully built all the packages using the i386 profile

Automation & Hooks

Last but not least let’s put everything we’ve done so far together to automate building two package sets. We can make use of cron(8) to schedule the tasks. Let’s add the first one to /etc/crontab like this:

0	23	*	*	7	root	env TERM=dumb /usr/local/bin/synth just-build /var/synth/pkglist-12.1-i386

What does it do? It will run synth at 11pm every sunday to build all of the packages defined in the package list referenced there. There’s two things to note here:

  1. You need to disable curses mode in all the profiles you’re going to use. Synth still expects to find the TERM environment variable to figure out the terminal’s capabilities. You can set it to dumb as done here or to xterm or other valid values. If you don’t set it at all, Synth will not run.
  2. The cron entry as we’re using it here will use the active profile for Synth. It’s better to explicitly state which profile should be used. Let’s add another line to crontab for building the amd64 packages for 13.0 on Friday night:
0	23	*	*	5	root	env TERM=dumb SYNTHPROFILE=LiveSystem /usr/local/bin/synth just-build /var/synth/pkglist-13.0-amd64

In general I’d recommend to consider not calling synth directly from cron but to write small scripts instead. You could for example backup the current package set before actually starting the new build or you could snapshot the dataset after the successful build and zfs-send it off to another system.

One last thing that you should be aware of is that Synth provides hooks like hook_run_start, hook_run_end, hook_pkg_failure and so on. If you’re considering using hooks, have a look at the Synth manpage, they are covered well there.

What’s next?

Next topic would be covering Poudriere. However I’m considering taking a little break from package building and writing about something else instead before returning to this topic.

FreeBSD package building pt. 4: (Slightly) Advanced Synth

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/freebsd_package_building_pt4.gmi

In the previous posts of this series, there was an introduction to package building on FreeBSD and we discussed basic Synth usage. The program’s configuration, working with compiler cache and using the tool to update the installed applications was covered as well.

This article is about some advanced functionality of Synth like using web reports, building package sets and serving repositories as well as taking a look at logs.

Reporting

Synth comes with very useful reporting reporting functionality. For the default LiveSystem profile, Synth writes its logs to /var/log/synth. There it also creates a subdirectory called Reports and puts an advanced web report in there. It looks like this:

% ls -1 /var/log/synth/Report 
01_history.json
favicon.png
index.html
progress.css
progress.js
summary.json
synth.png

We’re going to build and setup a web server to make it accessible. I will use OpenBSD HTTPd for a reason that we’re going to talk about in a minute (besides me liking it quite a bit). Let’s use Synth to install it first. Then we’re going to enable it and create a log directory for it to use:

# synth install www/obhttpd
# sysrc obhttpd_enable=YES
# mkdir /var/log/obhttpd

OpenBSD HTTPd successfully installed by Synth

Alright. Remember how I told you not to change Synth’s directory configuration unless you have a reason for it? Now we have: We’re going to serve both the web report and the packages over http and we’re using OpenBSD HTTPd for that. That browser is chrooted: Not just by default, BTW but by design! You cannot turn it off. Unless told otherwise, Synth has a directory structure that doesn’t fit this scenario well. So we’re going to change it.

First thing is creating a new directory for the logs and then changing the configuration so Synth uses that:

# mkdir -p /var/synth/www/log
> # synth configure

choc

Change setting E to /var/synth/www/log and save. The new directory will of course be empty. We could copy the old files over, but let’s just rebuild HTTPd instead. Using synth build or synth just-build doesn’t work, though; the tool will detect that an up to date package has already been built and that nothing needs to be done. That’s what the force command is handy for:

# synth force www/obhttpd

Web server setup for reports

Now that we have something to serve, we can edit the webserver’s configuration file. Simple delete everything and put something like this into /usr/local/etc/obhttpd.conf:

chroot "/var/synth/www"
logdir "/var/log/obhttpd"

server synth.local {
    listen on * port 80
    root "/log"
    log style combined
    location "/" {
        block return 302 "http://$HTTP_HOST/Report/index.html"
    }
}

This defines where the chroot begins and thus where the unprivileged webserver processes are contained. It also defines the log directory as well as a virtual host (simply called “server” in OpenBSD HTTPd) with the name “synth.local”. Either replace this with a proper domain name that fits your scheme and configure your DNS accordingly or use /etc/hosts on all machines that need to access it to define the name there.

The server part is pretty simple, too. It makes HTTPd bind on port 80 on every usable network interface. The web root is defined relative to the chroot, so in this case it points to /var/synth/www/log. I’ve grown a habit of using the more detailed combined log style; if you’re fine with the default common format, you can leave the respective line out. Finally the configuration block defines a special rule for location / which means somebody accesses the virtual host directly (i.e. http://synth.local in this case). It will make the browser be redirected to the report index instead. Getting a special file (like e.g. lang___python37.log in the log directory) will not trigger the rule and thus still work. This is just a convenience thing and if you don’t like it leave it out.

All that’s missing now is starting up the webserver:

# service obhttpd start

You should now be able to point your browser at the the vhost’s name (if you made it resolve). Just using the machine’s IP address is also going to work in this case since it’s the default vhost. But better make it reachable using the configured name as we’re adding another vhost in a moment.

Synth web report for the latest package run

Authentication

But for now what about security? Let’s say you don’t want to share your report with the whole world. One easy means of protecting it is by using HTTP basic auth. OpenBSD HTTPd uses standard .htpasswd files. These can however use various cryptographic hashes for the passwords – whereas HTTPd only supports one: bcrypt.

The first time I tried to do authentication with OpenBSD HTTPd, it drove me completely nuts as I couldn’t get it working. Fortunately I own Michael W. Lucas’ excellent book “Httpd and Relayd Mastery”. After digging it out the index indicated that I might want to read page 28. I did, banged my head against the table and immediately got it working using that hashing algorithm. Don’t be like me, skip trying to use foreign tools you may be used to and just do it right in the first place. HTTPd comes with its own htpasswd binary. Use that.

In this example I’m adding a user called “synth”. Use whatever you prefer. Then give the password two times. This leaves you with a valid .htpasswd file that HTTPd could use – if it was allowed to access it! Let’s fix that problem:

# chown root:wheel /var/synth/www/.htpasswd
> # chmod 0640 /var/synth/www/.htpasswd

Having the authentication information in place, we only need to add another location block to the webserver’s vhost configuration. Put the following in there after the line that closes the previous location block:

    location "/Report/* {
        authenticate with "/.htpasswd"
    }

Note the htpasswd file’s location! It’s within the chroot (or it couldn’t be accessed by the webserver), but outside the webroot directory. So HTTPd could never accidentally serve it to somebody who knew that it was there and requested the file.

The only thing that remains is restarting the webserver. Next time you visit the report page, you’ll be asked to authenticate first.

# service obhttpd restart

Package repository

So far all of our packages have been created in a directory outside of the webserver’s chroot. If we want to make them available via HTTP, we need to use another path for them. Therefore we’re going to create a directory and reconfigure Synth again:

# mkdir -p /var/synth/www/packages
# synth configure

This time it’s setting B. Change it to /var/synth/www/packages and save. Now let’s build a package that draws in a couple of dependencies:

# synth just-build chocolate-doom

We can watch it now via the web reports while it’s building. Since it’s a new directory where no packages exist, yet, Synth is first going to build the package manager again. During this early stage no report is available, but once that’s finished the reports work.

While we’re rebuilding all packages due to the new package directory, Synth can take advantage of ccache as we haven’t modified it’s path. Wonder how much of a difference that actually makes? Building llvm10 on its own, one time using the cache and one time (for testing purposes) without it will show you the difference:

Duration: 00:13:32 (with ccache)
Duration: 02:09:37 (uncached)

Synth web report while it’s building

It gives us all the information that the curses UI holds – and more. The number of entries for completed packages can be changed. You can browse those page-wise back to the very first packages. It’s possible to use filters to e.g. just list skipped or failed packages. You can view / download (whatever your browser does) the log files for all those packages. And there’s even a search (which can be very handy if you’re building a large package set).

Report with only 10 entries per page

As long as packages are being built, the report page also shows the builder information and automatically refreshes the page every couple of seconds. Once it completes, it removes builder info (which would only waste space) and stops the polling. You can always come back later and inspect everything about the latest package run. The next one will overwrite the previous information, though.

Synth report search function

Now that we have a bunch of newly built packages, let’s see what that looks like:

# ls -1 /var/synth/packages/All
autoconf-2.69_3.txz
autoconf-wrapper-20131203.txz
automake-1.16.3.txz
binutils-2.33.1_4,1.txz
bison-3.7.5,1.txz
ca_root_nss-3.63.txz
ccache-3.7.1_1.txz
celt-0.11.3_3.txz
chocolate-doom-3.0.1.txz
cmake-3.19.6.txz
curl-7.76.0.txz
db5-5.3.28_7.txz
docbook-1.5.txz
docbook-sgml-4.5_1.txz
docbook-xml-5.0_3.txz
docbook-xsl-1.79.1_1,1.txz
doom-data-1.0_1.txz
evdev-proto-5.8.txz
expat-2.2.10.txz
flac-1.3.3_1.txz
[...]

Showing only ignored packages in the report (none in this case)

The packages are there. But what’s in the main directory?

# ls -l /var/synth/www/packages
total 18
drwxr-xr-x  2 root  wheel  150 Jun  7 23:57 All
drwxr-xr-x  2 root  wheel    3 Jun  7 23:21 Latest

This is not a valid pkg(8) repository. Which is no wonder since we used just-build. So we’re going to have Synth create an actual repository from these packages next:

Searching in the report after the build was completed

# synth rebuild-repository
# ls -l /var/synth/www/packages
total 117
drwxr-xr-x  2 root  wheel    150 Jun  7 23:57 All
drwxr-xr-x  2 root  wheel      3 Jun  7 23:21 Latest
-rw-r--r--  1 root  wheel    163 Jun  8 00:02 meta.conf
-rw-r--r--  1 root  wheel    236 Jun  8 00:02 meta.txz
-rw-r--r--  1 root  wheel  40824 Jun  8 00:02 packagesite.txz

Here we go, that’s all that pkg(8) needs. Synth should have automatically updated your repository configuration to use the new location. Have a look at /usr/local/etc/pkg/repos/00_synth.conf – the URL should point to the new directory.

Serving the repository

The next step is to make the repository available in the network, too. So edit /usr/local/etc/obhttpd.conf once more and add another “server” (i.e. vhost):

server pkg.local {
    listen on * port 80
    root "/packages"
    log style combined
    location * {
        directory auto index
    }
}

One service restart later you should be able to access the repository via a web browser from any machine in the same subnet (if you got your DNS right):

# service obhttpd restart

Looking at the package repository with a browser

This is already it, but let’s prove that it works, too. I’m adding the “pkg.local” name to the machine’s 127.0.0.1 definition in /etc/hosts, then change the URL in the Synth repo to fetch packages via HTTP:

  url      : http://pkg.local,

I’ve also created a FreeBSD.conf to disable the official repository. Let’s stop the webserver for a second and then try to update the package DB:

# service obhttpd stop
# pkg update
Updating Synth repository catalogue...
pkg: Repository Synth has a wrong packagesite, need to re-create database
pkg: http://pkg.local/meta.txz: Connection refused
Unable to update repository Synth
Error updating repositories!

Ok, so there’s no other repository configured anymore and this one is not accessed via the filesystem. So we’re going to start the webserver once more (give it a sec) and then try again:

# service obhttpd start
# pkg update
Updating Synth repository catalogue...
Fetching meta.conf: 100%    163 B  0.2kB/s    00:01
Fetching packagesite.txz: 100%   40 KiB  40.8kB/s   00:01
Processing entries: 100%
Synth repository update completed. 148 packages processed.
All repositories are up to date.

Great! So now we can install DooM on this box or on any other machine running FreeBSD 13.0 which can reach it over the network.

Package sets

So far we’ve only either built all packages for what was already installed on the machine or for single ports that we selected at the command line. But now that we can serve packages over the network, it’s rather tempting to use a powerful build machine to build packages for various other FreeBSD machines, isn’t it? Let’s assume that you’re going to share packages with a KDE lover.

First we should prepare a list of packages that we need, starting with what is installed on our machine.

# pkg info | wc -l
345

Wow, that’s already quite some packages for such a pretty naked system! But we don’t need to consider them all as most of them are just dependencies. Let’s ask pkg(8) for the origin of all packages we explicitly installed (i.e. which were not recorded as automatically installed):

# pkg query -e %a=0 %o > /var/synth/pkglist
# cat /var/synth/pkglist
x11-wm/awesome
devel/ccache
graphics/drm-kmod
devel/git
www/obhttpd
ports-mgmt/pkg
ports-mgmt/portmaster
x11/sakura
x11/setxkbmap
security/sudo
ports-mgmt/synth
sysutils/tmux
x11/xfce4-screenshooter-plugin
x11/xorg-minimal

That’s better! But we certainly don’t need portmaster anymore, so we can take it off the list (and deinstall it actually). Let’s add www/firefox and x11/kde5 for our pal (and sort the list since it’s a bit ugly right now).

Once that’s done, we should be able to do a simple:

# synth build /var/synth/pkglist
Error: port origin 'devel/git' not recognized.
Perhaps you intended one of the following flavors?
   - devel/git@default
   - devel/git@gui
   - devel/git@lite
   - devel/git@svn
   - devel/git@tiny

Oh yes, right! We need to edit our list and specify the flavor to build! I’m going with the lite variant here, so the git line needs to be changed to this:

devel/git@lite

Then we can try again – and yes, it starts building after calculating the required dependencies.

Logs

Whoopsie! After about an hour stupid me removed the network cable for a moment. This has caused a couple of build failures (see screenshot). The report will display the phase that the build failed. In this case it’s the fetch phase (and we don’t have to look for the reason as we already know it). Sometimes a distfile mirror is temporary down or the distfile has been removed. In that case you will have to manually get the files and put them into the distfiles directory. Skipped ports also display the reason, i.e. which dependency failed previously.

Failed and skipped ports due to a connection problem

I better re-attach that cable right away and start the building over… Many hours later it has finished. But what’s this? Rust has failed again (and this time it wasn’t me)! And it failed at the stage phase. When this happens it’s usually because of a broken port got committed. Update your ports tree and hope that it has been fixed in the meantime. This is not the reason in our case, however.

Another phase, another failure!

But how do we find out what actually happened? Well, by looking at the logs, of course. Here’s the last 15 lines of the relevant log:

        finished in 183.141 seconds
  < Docs { host: TargetSelection { triple: "x86_64-unknown-freebsd", file: None } }
Install docs stage2 (Some(TargetSelection { triple: "x86_64-unknown-freebsd", file: None }))
running: "sh" "/construction/xports/lang/rust/work/rustc-1.51.0-src/build/tmp/tarball/rust-docs/x86_64-unknown-freebsd/rust-docs-1.51.0-x86_64-unknown-freebsd/install.sh" "--prefix=/construction/xports/lang/rust/work/stage/usr/local" "--sysconfdir=/construction/xports/lang/rust/work/stage/usr/local/etc" "--datadir=/construction/xports/lang/rust/work/stage/usr/local/share" "--docdir=/construction/xports/lang/rust/work/stage/usr/local/share/doc/rust" "--bindir=/construction/xports/lang/rust/work/stage/usr/local/bin" "--libdir=/construction/xports/lang/rust/work/stage/usr/local/lib" "--mandir=/construction/xports/lang/rust/work/stage/usr/local/share/man" "--disable-ldconfig"
install: creating uninstall script at /construction/xports/lang/rust/work/stage/usr/local/lib/rustlib/uninstall.sh
install: installing component 'rust-docs'
###  Watchdog killed runaway process!  (no activity for 78 minutes)  ###



--------------------------------------------------
--  Termination
--------------------------------------------------
Finished: Wednesday, 9 JUN 2021 at 00:04:47 UTC
Duration: 04:27:03

Ha! The build process was killed by the watchdog! Bad doggy? It does happen that the process would eventually have finished. Not this time. We have to dig a little deeper. In /var/log/messages of the build machine I can find the messages kernel: swap_pager: out of swap space and kernel: swp_pager_getswapspace(4): failed. This machine has 24 GB of RAM and 8 GB of swap space configured. And by building 6 huge ports concurrently, it exceeded these resources! Keep in mind that package building can be quite demanding, especially if you use tmpfs (which you should if you can).

So, there we are. We’ve configured our build server for web reports and serving the repository. We’ve looked at building package sets and covered a few examples of what can go wrong. And that’s it for today.

What’s next?

The last article about Synth will cover make.conf, signing repositories and using cron for automated builds. We’ll also take a brief look at profiles.

FreeBSD package building pt. 3: Intermediate Synth

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/freebsd_package_building_pt3.gmi

In this article we’ll continue exploring Synth. After covering a general introduction to package building as well as Synth’s basic operation in the previous articles, we’re going to look at the program’s configuration and using it for updating the system.

The first thing to do here is to get rid of the old ports tree and replace it with a newer one so that some updates become available:

# zfs destroy zroot/usr/ports
# zfs create zroot/usr/ports
# git clone --depth 1 -b 2012Q2 https://git.freebsd.org/ports.git /usr/ports
# mkdir /usr/ports/distfiles
# synth status

Synth regenerates the flavor index again, then gives the output. This time it will not just show all packages as “new” but most are either updates or rebuilds.

Synth status with new ports tree

Configuration

Let’s look at how you can change Synth’s behavior next:

# synth configure

Synth’s configuration is pretty straight-forward. Options A to G configure the various directories that the program uses. Should you change anything here? If you know what you are doing and have special needs: Maybe. If you actually ask the question whether to change any such directory, the answer is no. Option H (Compiler cache) is a special one. We’ll get to that in a minute.

With I you set the number of builders that Synth uses. J means how many threads each builder may use; think make -j $SOMENUMBER. When compiling a single piece of software, it’s usually recommended to set the number of jobs equal to the machines core count + 1. Take a look at the screenshot above: Doing the math, we’re configuring for 24 cores here – on a system that has 8 (with Hyper Threading).

Synth’s configuration menu

Why does Synth choose to over-provision the resources so much? The reason is simple: It’s only over-provisioned when more than three builders are active at the same time. Often enough not all builders will be at the build stage (where the compiling happens) at the same time. Most other stages are much lighter on CPU – which would mean that you’re wasting available resources (and thus prolong the total build time). Also in the previous post you’ve seen that LLVM and Rust took hours to build (all four remaining builders were idle most of the time!). If the cap would have been lower, build times would have increased even more.

So what’s the best setting for builder count and max jobs? There’s no single answer to that. It depends on both your machine and on the set of ports that you build. Play with both values a bit and get a feeling what works best for you. Or leave it at the default that Synth calculated for your system which will probably work well enough.

Options K and L are real speed boosters. They control whether the base system for the builder (“localbase”) and the directory where the program is built are using tmpfs or not. Tmpfs is a memory-backed filesystem. If you disable one or both options, compilation times increase a lot because all the data that needs to be copied on setting up a builder and compiling software will be written to your disk. For an extreme example: On one of my machines, building and testing a port increased from slightly over 30 seconds with tmpfs to over 4 minutes (!) without it. Yes, that machine uses an HDD and it was occupied with other things besides building packages. But there is more than a negligible impact if you disable tmpfs.

So when to do it in the first place? If you’re on a machine with little RAM you might have to do this. Some ports like LLMV or Firefox require lots of RAM to build. If your system starts swapping heavily, disable tmpfs. Ideally build those ports separately and leave tmpfs on for all the rest.

Then there’s option M which toggles the fancy colored text UI on or off. If you turn it off, you’ll get a simple monochrome log-like output of which builder started or finished building which port. Every now and then the info that’s in the top bar of the UI (e.g. number of packages completed, number of packages remaining, skips, etc) gets printed.

Finally we have option N which toggles using pre-built packages on or off. It’s off by default which means that Synth will build everything required for the selected package set. If you enable this option and have e.g. the official FreeBSD repository enabled as well, it will try to fetch suitable packages from there that can be used as the buildtime or runtime dependencies of the packages that still need to be built. If you’re mixing official and custom packages this could be a huge time saver. I admit that I have never used this option.

And what’s this profile thing? Well, ok. The configuration we’ve been looking at is for the default profile. You can create additional ones if you want to. And that’s where the directories that I told you to ignore come into play. You could for example create a profile to use a different ports tree (e.g. the quaterly branch) and switch between the profiles to build two different packages sets on one machine. While Synth can do this, that is the point where I’d advise you to try out Poudriere instead. The main benefit of Synth over Poudriere is ease of use and when you’re trying to do clearly advanced things with Synth you might as well go for the officially supported FreeBSD package builder instead.

Compiler cache

Let’s disable the curses UI just to see what Synth looks like without it and save the config change. If you plan to build packages regularly on your system, you will definitely want to set up the compiler cache. To be able to use it, we first need another package installed, though: ccache. We’re going to build it and then install it manually this time:

# synth just-build devel/ccache
# pkg add /var/synth/live_packages/All/ccache-3.7.1_1.txz

Then we’re going to create a directory for it and go back to Synth’s configuration menu:

# mkdir -p /var/tmp/ccache/synth
# synth configure

Now change H to point to the newly created directory and save. Synth will use ccache now. But what does it do? Ccache caches the results from compilation processes. It also detects if the same compilation is happening again and can provide the cached result instead of actually compiling once again. This means that the first time you compile it doesn’t make a difference but after that the duration of building packages will drop significantly. The only cost of this is that the cached results take up a bit of space. Keep ccache disabled if drive space is your primary concern. In all other cases definitely turn it on!

Ccache directory and config after the system upgrade

Updating the system

Next is using Synth to update the system.

# synth upgrade-system

After determining which packages need to be built / rebuilt, Synth will start doing so. We’ve turned off the text UI, so now we get only the pretty simplistic output from standard text mode.

Package building in pure text mode

It provides you with the most important information about builders: Which builder started / finished building which package when. You don’t get the nice additional info about which state it’s in and how long it has been busy building so far. As mentioned above, Synth will print a status line every couple of minutes which holds the most important information. But that’s all.

Status lines amidst the builder output

What happens if something goes wrong? I simulated that by simply killing one of the processes associated with the rust builder. Synth reports failure to build rust and prints a status line that shows 17 package skips. In contrast to the curses UI it does not tell you explicitly which ones were skipped, though!

Simulated package build failure

When Synth is done building packages, it displays the tally as usual and does some repository cleanup by removing the old packages. Then it rebuilds the repository.

Tally displayed after completion and repository cleanup

Since we asked Synth to upgrade the system, it invokes pkg(8) to do its thing once the repository rebuild is complete.

Package repository rebuilt, starting system upgrade

And here’s why I strongly prefer prepare-system over upgrade-system: The upgrade is initiated whether there were failed packages or not. And since pkg(8) knows no mercy on currently installed programs when they block upgrades, it will happily remove them by default! To be safe it makes sense to always review what pkg(8) would do before actually letting it do it. Yes, it’s an additional step. Yes, most of the time you’ll be fine with letting Synth handle things. But it might also bite you. You’ve been warned.

System upgrade in progress – it may remove packages!

Every now and then you may want to run synth purge-distfiles (unless you have unlimited storage capacity, of course). It will make the tool scan the distinfo files of all ports and then look for distfile archives of obsolete program versions to remove.

Cleaning up old distfiles with purge-distfiles

There might not be a large gain in our example case, but things do add up. I’ve had Synth reclaim multiple GB on a desktop machine that I regularly upgraded by building custom packages. And that’s definitely worth it.

What’s next?

The next article will cover some leftover topics like logs, web report and repository sharing.

FreeBSD package building pt. 2: Basic Synth

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/freebsd_package_building_pt2.gmi

The previous article was an introduction into package building on FreeBSD in general. It also included references to various other articles about package management and working with ports. Familiarity with those topics is assumed here. If you don’t feel confident, have a look there and do a bit of reading first. Or just go ahead and do some research if you hit something that you have trouble understanding – package building is not a beginner’s topic but it ain’t rocket science, either.

In the following section I’m following a “naive approach” of just installing and exploring Synth on a typical desktop system. Afterwards I’m switching to the test system that was described in the previous post so that I can show updating, too.

Synth basics

So let’s get started with Synth now, shall we? The first thing to do is of course installing it:

# pkg install synth

If you want to or need to build it from source instead, the origin is ports-mgmt/synth. It has one compile-time option: Enabling or disabling the watchdog feature. By default it is enabled which means that Synth is supervising the build processes and if one looks like it has stalled, the watchdog will kill it. When you are building on very slow hardware (why are you doing this in the first place?) it may be an anti-feature that causes very complex ports (e.g. Firefox) to be killed even though it would otherwise eventually finish. In that case you can disable the watchdog option and rebuild Synth. Or you go to the junkyard and get some more capable hardware!

Synth port options

As with every new piece of software, it’s probably a good idea to get an overview of what it can do:

% synth help
Summary of command line options - see synth.1 man page for more details
===============================================================================
synth status              Dry-run: Shows what 'upgrade-system' would build
synth configure           Brings up interactive configuration menu
synth upgrade-system      Incremental rebuild of installed packages on system.
                          Afterwards, the local repository is rebuilt and the
                          system packages are automatically upgraded.
synth prepare-system      Like 'upgrade-system' but ends when repo is rebuilt
synth rebuild-repository  Rebuilds local Synth repository on command
synth purge-distfiles     Deletes obsolete source distribution files
synth status-everything   Dry-run: Shows what 'everything' would build
synth everything          Builds entire ports tree and rebuilds repository
synth version             Displays version, description and usage summary
synth help                Displays this screen
synth status [ports]      Dry-run: Shows what will be rebuilt with given list
synth build [ports]       Incrementally build ports based on given list, but
                          asks before updating repository and system
synth just-build [ports]  Like 'build', but skips post-build questions
synth install [ports]     Like 'build', but upgrades system without asking
synth force [ports]       Like 'build', but deletes existing packages first

synth test [ports]        Just builds with DEVELOPER=yes; pre-deletes pkgs

[ports] is a space-delimited list of origins, e.g. editors/joe editors/emacs.
It may also be a path to a file containing one origin per line.

That’s a nice list and even better: The commands are explained in a way that mere mortals can understand. Basically it can compare installed program versions to those in ports for you with synth status. You can configure synth with synth configure (we’ll look at this later, the default config is fine for now). And it can bulk-build packages and use those to upgrade the system for you with synth upgrade-system. And those are the important ones that you’ll need to begin with.

There’s also synth prepare-system in case you want Synth to only build the packages (maybe you want to do the upgrade manually later). With synth rebuild-repository you can make the tool, well, rebuild the repository from the currently built packages. Since it does this after finishing package builds, you will need it only if you cancelled building somewhere in the middle and want it to rebuild the repo anyway. The synth purge-distfiles command is useful if you use Synth for a while. It will scan for no longer needed distfiles (for obsolete program versions) and potentially free quite a bit of space for you.

And that’s it. The synth status-everything and synth everything commands are only useful if you want to build packages for the entire FreeBSD ports collection. But that’s certainly not basic usage! The various options that act on a single port or a list of ports is also advanced usage. They’ll come in handy if you plan on developing ports, build packages for other machines to use or have special needs. If you plan to keep your machine’s installed packages consistent for production use, know that these commands exist but stay away.

Package status

So much for some theory and on to actually doing something. Let’s ask Synth about the status of our packages:

# synth status
It seems that a blank PORTSDIR is defined in /etc/make.conf
Configuration failed to load.

Alright, this message can be a little confusing as by default there is no such file in FreeBSD! What Synth actually wants to tell you is that it cannot find a ports tree on the system and thus cannot do anything. So let’s get the ports in place right now:

# portsnap auto

With the ports tree available on our system we can try again:

# synth status
Configuration invalid: [C] Distfiles directory: /usr/ports/distfiles
Rather than manually creating a directory at this location, consider
using a location outside of the ports tree. Don't forget to set
'DISTDIR' to this new location in /etc/make.conf though.

This time the problem is pretty straight-forward: The distfiles directory does not exist. Synth gives us some advice in case we want to use a custom location, however I prefer the standard one.

# mkdir /usr/ports/distfiles
# synth status
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
 progress: 3.32%

Depending on your build machine this can take anything from a couple of minutes to a quarter of an hour or so. Before FreeBSD introduced port flavors, it was a matter of seconds, but for a few years now Synth has to do some additional work up-front. When it’s done it will print something like this:

Querying system about current package installations.
> Installed package ignored, devel/py-evdev package unmatched
> Installed package ignored, devel/py-pyudev package unmatched
> Installed package ignored, devel/py-six package unmatched
> Stand by, comparing installed packages against the ports tree.
> Stand by, building pkg(8) first ... done!
> These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
>   N => print/indexinfo
>   N => devel/gettext-runtime
> [...]

For the first run, Synth has to build all of the packages. Therefore all are marked new. But what’s that “Installed package ignored” thing? That usually happens if a package is installed on the system that was built from a port that does no longer exist in the new tree. Sometimes it’s also weirdness that can happen with flavored ports. Disregard this if it’s only affecting packages that you don’t want to use directly. The correct ones will be pulled in as dependencies anyway. Now let’s build our first package set:

# synth prepare-system

After gathering some information, Synth will start building the packages that are currently installed on this machine (which for simplicity’s sake resemble those of the test system). Let’s have a look:

Synth started building packages

By default, Synth shows a nice curses-based text UI that displays a lot of information (see screenshot above). It shows the total count of packages to build, the number of packages that remain to be built and the count of already built packages. In case a port does not successfully build on this system (update ports and try again – if the problem persists either file a bug report or contact the maintainer), Synth displays that, too. Ignored packages are those that don’t work on your particular system; maybe the application is known to not build on the version of FreeBSD you are using or not on your architecture (if for example you’re building on ARM). Skipped count goes up if a failed port was a dependency for others. Obviously Synth cannot build them.

Synth also displays the current system load: A load of e.g. 2.0 means that two cores of your CPU are completely busy. If your CPU supports Hyper-threading that basically doubles the available core count. A load higher than 8.0 on a 4 core system with HT means that the system has currently more work than it’s able to fulfill concurrently. It’s not ideal but it’s not something to worry too much about, either. Watch swap usage. A little bit of swapping for large ports is not much of a problem. But if your system swaps a lot that will slow down the package building a lot. Should you manage to run out of swap, package builds will fail. You can adjust Synth’s configuration if you’re unhappy with either load or swap usage. But we’ll get to that.

Synth after about 20 minutes

There’s also the number of packages built per hour and the impulse as well as the elapsed time. Initially packages per hour and impulse are identical because the two are similar. However the former is the average number of packages built over the whole build time while the latter is the number of packages built within the last 500 seconds.

But that’s only the top bar. The next part of the UI is for the builders. Builders are clean build environments (“chroot jails”) that are created before building a package and then teared down afterwards. The next package gets a new fresh environment. The number of builders means how many packages can be building concurrently. On the screenshot you can see 6 builders which are color-coded to be easier to distinguish. This may look different on your machine and here’s why: Synth tries to guess reasonable defaults for your machine. If you have an old dual-core PC, it will use less than the six builders it deemed right for the quad-core i7 that I used as my test machine here. Expect to see a much higher number of builders on modern servers with a higher core count.

Idle builders after 5 hours

For each builder you see how long it has been working already, which phase it is currently in (remember the various build targets of the ports infrastructure?) which port the builder is occupied with and how many lines of log output the builder has has produced so far. Especially the last information is not going to help you a lot when you begin building packages. After a while you’ll know roughly how many lines of output some of the big ports like LLVM produce and can judge if it’s going to take another two hours to finish or more like half an hour. (And then a new version of LLVM comes out which takes even longer to build so that your previous idea of “how many lines” is no longer valid. That’s how things go.)

And finally there’s the rest of the screen made up of a list of recently finished packages. If you take a look at the second screenshot, you’ll see some ports where the origin ends with @py37. Here the builder is busy building a flavored port – and it’s building for Python 3.7. What’s @native you ask? Well, Python ports are a typical example of flavored ports but they are not the only ones. The binutils port for example is able to be built as part of a native toolchain or a cross toolchain in case you want to e.g. cross compile packages for riscv on your much more powerful amd64 machine.

First builder has shut down

What’s the deal with idle builders like those on screenshot 4? Idle builders are the ones that Synth has already prepared for another package to build in but has not been able to use so far. Take a look at screenshot 4: There’s 53 more packages to build but only two builders are occupied with work while four are idle. Can you guess why? The two ports currently building are LLVM10 and Rust. And the “problem” is that all other 51 packages that are still on the list depend on either LLVM or Rust directly or indirectly! So while there’s more work for the builders, Synth has to wait for the dependencies to finish building before it can start building the remaining packages.

At some point no further builders will be required. In that case Synth shuts them down (see screenshot 5).

Last package build starting after about 6 hours

When Synth has finished building all the packages requested, it will present you with the final statistics. Then it cleans up the package directory, removing obsolete old packages. Eventually it will build a valid pkg(8) repository for you – and then it’s finally done.

Synth has completed its task

Should you ever want to quit a package run early, try to avoid CTRL-C. If you press CTRL-Q instead, Synth will shutdown gracefully. This means that now new builders will be started and the tool exits properly once those that are already running complete.

Pkg(8) repositories

I’m covering the case where you want to use your own package repository instead of the official FreeBSD one. If you want to use both, make sure you read what I pointed my readers to in the previous article. Then configure your new repository as I do here but simply don’t disable the official repo.

The standard FreeBSD repository is configured in /etc/pkg/FreeBSD.conf. Do not make changes there, though! This file belongs to the base system and upgrades may overwrite it. If you want to change the settings, create /usr/local/etc/pkg/repos/FreeBSD.conf (and the missing directories). The latter file will override the former. In case you just want to disable the official package repository, simply put this single line into the file:

FreeBSD: { enabled: no }

Synth should automatically generate a config file there if you use it to upgrade the system. The file is called /usr/local/etc/pkg/repos/00_synth.conf and has the following content:

# Automatically generated.

Synth: {
  url      : file:///var/synth/live_packages,
  priority : 0,
  enabled  : yes,
}

Now you only need to update the repository information:

# pkg update

And that’s it. Pkg(8) will now happily use your local package repository.

What’s next?

The next article will feature Synth configuration, upgrading and advanced usage.

FreeBSD package building pt. 1: Introduction and test system

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/freebsd_package_building_pt1.gmi

In 2017 I started to post a series of well-received articles that was meant to culminate in a tutorial on building packages on FreeBSD. I wrote about the history of package managers, about how to use FreeBSD’s pkg(8) package manager (part 1 and part 2) as well as an introduction to working with ports (part 1 and part 2).

Then I had to stop for technical reasons (after a major change to the ports infrastructure, the tool that I wanted to write about had not been updated to work with the new ports tree, yet!). In 2019 I eventually found the time to publish two more articles that were originally meant to come right after the 2017 posts. They covered using classic tools for ports management (part 1 and part 2).

They were meant to be the stepping stone to what I actually wanted to cover: Package building! The second article ended with:

I’ve planned for two more articles that will cover building from ports the modern way(tm) – and I hope that it will not take me another two years before I come to it…

That was… Yikes! In fall 2019… So I’d better hurry up now. I had hinted that using ports directly was not the recommended thing to do anymore and that while you should use packages unless you need ports, you really should use packages even in the latter case! And that’s what we’re going to do in the next few articles.

Why roll your own packages?

There are valid reasons to not use FreeBSD’s official packages:

  • Most frequently you need build-time options configured differently.
  • Or your organization uses a different default version of a scripting language (Perl, Python, Ruby, …) that what FreeBSD chose at the time being.
  • Maybe you’re running a customized version of FreeBSD that’s not officially supported.
  • Perhaps you need programs where the license forbids binary redistribution.
  • Or you use custom ports for some programs that of course cannot be available in official packages then.

Things like that. People choose ports when they need more control over their applications. There are good reasons to avoid using ports the traditional way, too, however:

  • It takes time and resources to build everything from source; especially if you do this on multiple machines, it’s a waste.
  • It makes updates much more complicated (see the second 2019 post mentioned above).
  • It clutters your system with build-time dependencies that need to be installed first so that the actual programs can be built.
  • Depending on what other software your machine has installed, the resulting programs might differ from that of other machines even if they built the same port!

While the former two points are mostly relevant if you manage multiple machines, I’d recommend rolling your own packages even for the single FreeBSD workstation that you use – if the official packages don’t suit you. Let me state this again: Try to go with the packages provided by the FreeBSD project first. Build your own packages if you have to (educating yourself is a completely valid reason to do it anyway, though).

Package builders

When you decide to roll your own packages, you have two options: Synth, the much easier and nicer package builder and Poudriere, the advanced build tool that FreeBSD uses and provides official documentation for.

Which one should you choose? I’m going to show how to work with both so you can compare them and make an informed decision. If you’re just getting started, you may want to give Synth a try. It is also a good choice when you use DragonFly BSD, too: The dsynth tool that they have in base was inspired by Synth (and if it wasn’t written in Ada they certainly would just have imported it instead of creating a re-implementation in C). You should also know that the Synth project is in maintenance mode. Its author still fixes bugs and takes pull requests on GitHub, but it’s feature-complete.

The main advantage of Synth is that it’s dead simple to setup and use, too, whereas Poudriere is a somewhat complex beast. Synth also shines when you want to use it to keep one machine up to date with packages as it can do those updates for you. Poudriere on the other hand allows you to do things like maintaining package repositories for multiple versions of FreeBSD as well as multiple architectures from one build machine. If you need that, forget Synth.

Ports and Git

One major change that was made in FreeBSD since the previous article was published is that the project migrated to using Git repositories. FreeBSD development started on CVS but in 2008 the src repository was successfully migrated to using Subversion. In mid 2012 docs and ports also made the switch. Subversion has been used ever since until December 2020 when doc and src transitioned to Git. After some remaining issues were solved, ports also migrated to Git in April 2021. While src changes get pushed back to Subversion for FreeBSD’s branches of 11 and 12, when it comes to ports, an era has ended.

Get rid of that habit of visiting svnweb.freebsd.org and start getting used to working with cgit.freebsd.org instead.

If you are unsure of your Git skills, you may want to at least skim over the Git-primer in FreeBSD’s documentation.

At least get familiar with the concepts. You should for example know that Git is able to do things like a shallow clone; looking things up when you need them is no problem but not being aware of them at all is.

While both Subversion and Git are used for version control and both use repositories, they are fundamentally different. Subversion is the best known version control system of the second generation (so-called (centralized) “networked VCS”). Git is the most popular one of the third generation (“decentralized VCS”).

If you use Subversion, there’s a central repository somewhere and you checkout a certain revision of the versioned files from there. The files that were checked out form your working directory. In Git you clone a remote repository and thus receive all the data needed to get a local copy of the whole repo. The files for the working directory are checked out from the local copy. The major difference is: You have the whole history available locally.

In case you have no need for all that data, you do a shallow clone instead of a regular full clone. To give you an example: If you do a shallow clone of the current ports tree today, the result is about 840 MB in /usr/ports – of which 85 MB in size is the Git repository. A full clone is about 1.7 GB in size of which about 920 MB is for the repo! So if you don’t need the history, save some space on your drive and save some donated bandwidth of the FreeBSD project.

Considerations before you start

While you can certainly start rolling your own packages on a fresh system, it’s also fine to begin doing so on a system that has used the standard FreeBSD ports or packages so far. There’s nothing wrong with that actually. Taking the opposite way and going back from custom packages to the official ones is also possible, of course. That latter case requires some more planning, though. Think about why you started building your own packages in the first place, find out if the standard packages fit your current needs and think about the consequences. If you’re uncertain, you may want to start over with regular packages. If you want to make the switch back, it’s best to re-install all the packages (use the flag -f with pkg upgrade). If you’re using ZFS in a way that supports Boot Environments, create one first.

In fact you can even choose to use both official and custom packages on the same machine: Instead of building the full set of packages that you need, you build just the packages yourself that you need to customize and use the packages from the standard FreeBSD package repositories for the rest. This works by configuring your package repository as an additional one for pkg(8) and giving it a higher priority. If you consider doing this and using multiple package repositories, be sure to

% man 5 pkg.conf

first. It doesn’t hurt to skim over the whole manpage, but make sure that you at least read the section REPOSITORY CONFIGURATION. I used that approach on a weaker machine in the past, but wouldn’t generally recommend it. If your hardware is recent enough, you can compile everything yourself. Otherwise it might make more sense to build on a somewhat beefy system and distribute the packages to your other machine(s). But that’s only me. Maybe mixing packages is the right solution for your use case.

Building a test system

The rest of this post is about building a test system to simulate an environment with pre-installed packages. I do this so that I can show off a few things regarding updates in the next post. If you want to follow along, make sure that you have Git installed and that your directory or dataset /usr/ports as well as /usr/src is empty. I’m assuming that you’re using FreeBSD 13.0.

The first thing to do is getting the ports tree in place. I’m doing a shallow clone of the 2021 first quarter branch (so I can update later). Then I clone operating system source repository as well:

# git clone --depth 1 -b 2021Q1 https://git.freebsd.org/ports.git /usr/ports
# git clone --depth 1 -b releng/13.0 https://git.freebsd.org/src.git /usr/src

Now we don’t need the packages anymore and forcefully remove them all (including pkg). The reason for this is that we want to build the older versions of the old ports tree that we just cloned.

# pkg delete -af

For convenience we’re going to build all the required programs using portmaster as discussed in a previous article (see the beginning of this post). Of course we need to build and install it first:

# make -C /usr/ports/ports-mgmt/portmaster install clean

Alright. Now we’re building some leaf ports to simulate a very simple development system. These ports draw in a number of dependencies so that we end up with about 350 packages (which is still a very low count for a desktop system). I’m building tmux and sudo simply because I always need to have them available and git just because we needed it before anyway. The rest is the graphics drivers, a simple subset of X11 + the command to set a different keyboard layout as well as the awesome window manager and the simple but nice GTK+ terminal emulator called sakura. Since I’ll need to take screenshots for the upcoming posts, I figured that I might include an application for that, too.

# portmaster -dG sysutils/tmux security/sudo devel/git graphics/drm-kmod x11/xorg-minimal x11/setxkbmap x11-wm/awesome x11/sakura x11/xfce4-screenshooter-plugin

And that’s it for today.

What’s next?

Next station is taking a look at building packages and updating the system using Synth.

My first FreeBSD port: Castor

Summary

My first port has been committed to the FreeBSD port’s collection. It’s Castor, a graphical browser for text-based Internet protocols (currently: Gemini, Gopher, Finger). I write about the porting and about the actual application.

Article

[New to Gemini? Have a look at my Gemini FAQ.]

The actual post can be found here (if you are using a native Gemini client):

gemini://gemini.circumlunar.space/users/kraileth/neunix/2020/new_port_castor.gmi

If you want to use a webproxy to read it from your web browser, follow this Link.

Using FreeBSD with Ports (2/2): Tool-assisted updating

In the previous post I explained why sometimes building your software from ports may make sense on FreeBSD. I also introduced the reader to the old-fashioned way of using tools to make working with ports a bit more convenient.

In this follow-up post we’re going to take a closer look at portmaster and see how it especially makes updating from ports much, much easier. For people coming here without having read the previous article: What I describe here is not what every FreeBSD admin today should consider good practice (any more)! It can still be useful in special cases, but my main intention is to discuss this for building up the foundation for what you actually should do today.

Building a desktop

Last time we stopped after installing the xorg-minimal meta-port. Let’s say we want a very simple desktop installed on this machine. I chose the most frugal *nix desktop, EDE (the Equinox Desktop Environment, looking kind of like Win95), because that’s drawing in things that I need for demonstrating a few interesting things here – and not that much more.

Unfortunately in the ports tree that we’re using, exactly that port is broken (the newer compiler in FreeBSD 11.2 is more picky than the older ones and not quite happy with EDE code). So to go on with it, we have to fix it first. I’ve uploaded an additional patch file from a later version of the port and also prepared a patch for the port’s Makefile. If you want to follow along, you can just copy the three lines below to your terminal:

# fetch http://www.elderlinux.org/files/patch-evoke_Xsm.cpp -o /usr/ports/x11-wm/ede/files/patch-evoke_Xsm.cpp
# fetch http://www.elderlinux.org/files/patch_ede_port -o /usr/ports/x11-wm/ede/patch_ede_port
# patch -d /usr/ports/x11-wm/ede -i patch_ede_port

Using Portmaster to build and install EDE

Thanks to build-time dependencies and default options in FreeBSD it’s still another 110 ports to build, but that’s fine. We could remove some unneeded options and cut it down quite a bit. Just to give you an idea: By configuring only one package (doxygen) to not pull in all the dependencies that it usually does, it would be just 55 (!) ports.

But let’s say we’re lazy. Do we have to face all of those configure dialogs (72 in cause you are curious)? No, we don’t. That’s why portmaster has the -G flag which skips the config make target and just uses the standard port options:

# portmaster -DG x11-wm/ede

EDE was successfully installed

Using this option can be a huge time-saver if you’re building something where you know that you don’t need to change the options for the application and its dependencies.

System update

Now that we have a simple test system with 265 installed but outdated packages. Let’s update it! Remember that unlike e.g. Linux, FreeBSD keeps third party software installed from packages or ports and the actual operating system separate. We’ll update the latter first:

# freebsd-update upgrade -r 11.3-RELEASE

With this command, we make the updater download the required files for the upgrade from 11.2-RELEASE to 11.3-RELEASE.

Upgrading FreeBSD to 11.3-RELEASE

When it’s done, and we’ve closed the three lists with the removed, updated and new files, we can install the new kernel:

# freebsd-update install

Once that’s done, it’s time to reboot the system so the new kernel boots up. Then the userland part of the operating system is installed by using the same command again:

# shutdown -r now
# freebsd-update install

Kernel upgrade complete

Preparations

Now in our fresh 11.3 system we should first get rid of the old ports tree to replace it with a newer one, right? Wait, hold that rm command for a second and let me show you something really useful!

If you take a look at the /usr/ports directory, you’ll find a file appropriately named UPDATING. And since that’s right what we were about to do, why not take a look at it?

So what is this? It’s an important file for people updating their systems using ports. Here is where ports maintainers document breaking changes. You are free to ignore it and the advice that it gives and there’s actually a chance that you’ll get away with it and everything will be fine. But sometimes you don’t – and fixing stuff that you screwed up might take you quite a bit longer than at least skimming over UPDATING.

# less /usr/ports/UPDATING

But right now it’s completely sufficient to look at the metadata of the first notification which reads:

20180330:
  AFFECTS: users of lang/perl5*
  AUTHOR: mat@FreeBSD.org

The main takeaway here is the date. The last heads-up notice for our old ports tree was on 2018-03-30.

Checking out a newer ports tree

Now let’s throw it all away and then get the new ports tree. Usually I’d use portsnap for this, but in this case I want a special ports tree (the one that would have come with the OS if I got ports from a fresh 11.3 installation), so I’m checking it out from SVN:

# rm -rf /usr/ports/.* /usr/ports/*
# svnlite co svn://svn.freebsd.org/ports/tags/RELEASE_11_3_0 /usr/ports

If you’re serious about updating a production server that you care about, now is the time to read through UPDATING again. Search for the date string that you previously took a note of and then read the messages all the way up to the beginning of the file. It’s enough to read the AFFECTS lines until you hit one message that describes a port which you are using. You can ignore all the rest but should really read those heads-up messages that affect your system.

What software can be updated?

BTW… You know which packages you have installed, don’t you? A huge update like what we’re facing here takes some planning up-front if you want to do it in a professional manner. In general you should update much more often, of course! This makes things much, much easier.

Updating from ports

Ok, we’re all set. But which software can be updated? You can ask pkg(8) to compare installed packages to the respective distinfo from the corresponding port:

# pkg version -l "<"

If you pipe that into wc -l you will see that 165 of the 265 installed packages can (and probably should) be updated.

Updating software from ports

We’ll start with something really simple. Let’s say, we just want to update pkgconf for now. How do we do that with portmaster? Easy enough: Just like we would have portmaster install it in the first place. If something is already installed, the tool will figure out if an update is available and then offer to update it.

# portmaster -G devel/pkgconf

And what will happen if the port is already up to date? Well, then portmaster will simply re-build and re-install it.

Partial update finished

While partial updates are possible, it’s a much better idea to keep the whole system updated (if at all possible). To do that, you can use the -a flag for portmaster to update all installed software.

Another nice flag is -i, which is for interactive mode. In this mode, portmaster will ask for every port if it should be updated or not. If you’re leaving out critical ports (dependencies) this can lead to an impossible update plan and portmaster will not start updating. But it can be useful for cherry-picking.

Interactive update mode

Now let’s attempt to upgrade all the ports, shall we?

# portmaster -aG

As always, portmaster will show you its plan and ask for your confirmation. There are two things that probably deserve a word or two about them. Usually the plan is to update an application, but sometimes portmaster wants to re-install or even downgrade them (see picture)! What’s happening here?

Re-installs mostly happen when a dependency changed and portmaster figured out that it might be a good idea to rebuild the port against the newer version of the dependency, even though there is no newer version available for the actual application.

Upgrading, “downgrading”, re-installing

Downgrades are a different thing. They can happen when you installed something from a newer ports tree and then go back to an older one (something you usually shouldn’t do). But in this case it’s actually a false claim. Portmaster will not downgrade the package – it was merely confused by the fact that the versioning scheme changed (because of the 0 in 2018.4 it thinks that this version is older than the previous 2.1.3…).

Moved ports

If you’re paying close attention to all the information that portmaster gives you, you’ll have seen lines like the following one:

===>>> The x11/bigreqsproto port moved to x11/xorgproto

There’s another interesting file in the ports tree called MOVED. It keeps track of moved or removed ports. Sometimes ports are renamed or moved to another category if the maintainer decides it fits better. Portmaster for example started as sysutils/portmaster and was later moved when the ports-mgmt category was introduced. However you won’t find this information in the MOVED file – because it happened before the time that the current MOVED keeps records for (i.e. early 2007 in this case).

The example above is due to the fact that last year the upstream project (Xorg) decided to combine the protocol headers into one distribution package. Before that there were more than 20 separate packages for them (and before that, once upon a time, all of Xorg had been one giant monolithic release – but I digress…)

Problem with merged ports

The good news here is that portmaster is smart enough to parse the MOVED file and figure out how to deal with this kind of changes in the ports tree! The bad news is that this does not work for more complicated things like the merges that we just talked about…

So what now? Good thing you read the relevant UPDATING notification, eh?

20180731:
  AFFECTS: users of x11/xorg and all ports with USE_XORG=*proto
  AUTHOR: zeising@FreeBSD.org

Bulk-deleting obsolete ports and trying again

So let’s first get rid of the old *proto packages with the command that developer Niclas Zeising proposes and then try again:

# pkg version -l \? | cut -f 1 -w | grep -v compat | xargs pkg delete -fy
# portmaster -aG

Required options

Alright, we have one more problem to overcome. There are ports that will fail to build if we run portmaster with the -G flag. Why? Because they have mandatory ports options where you need to choose from things like a backend or a certain mechanic.

The “mandatory options” case

One such case is freetype2. Since this one fails, build it separately and do not skip the config dialog for this one:

# portmaster print/freetype2

Once that’s done, we can continue with updating all the remaining ports. After quite a while (because LLVM is a beast) all should be done!

Updating complete!

Default version changes

Did you read the following notice in UPDATING?

20181213:
  AFFECTS: users of lang/perl5*
  AUTHOR: mat@FreeBSD.org

For the big update run we ignored this. And in fact, portmaster did update Perl, but only to the latest version in the 5.26 branch of the language:

# pkg info -x perl
perl5-5.26.3

Why? Well, because that was the version of Perl that was already installed. Actually this is ok, if you’re not using Perl yourself and thus can live without the latest features added. However if you want to (or have to) upgrade to a later Perl major version we have a little bit more work to do.

First edit /etc/make.conf and put the following line in there:

DEFAULT_VERSIONS+=perl5=5.28

This is a hint to the ports framework that whenever you’re building a Perl port, you want to build it against this version. If you don’t do that, you will receive a warning when building the other Perl. Because in this case you’re installing an additional Perl version but all the ports will use the primary one. So more likely than not you don’t want to do this.

Upgrading Perl

Next we need to build the newer Perl. The special thing here is that we need to tell portmaster of the changed origin so that the new version actually replaces the old one. We can do this by using the -o flag. Mind the syntax here, it’s new origin first and then old origin and not vice versa (took me a while to get used to it…)!

But let’s check the origin real quick, before we go on. The pkg command above showed that the package is called perl5. This outputs what we wanted to know:

# pkg info perl5
perl5-5.26.3
Name           : perl5
Version        : 5.26.3
Installed on   : Tue Sep 10 21:15:36 2019 CEST
Origin         : lang/perl5.26
[...]

There we have it. Now portmaster can begin doing its thing:

# portmaster -oG lang/perl5.28 lang/perl5.26

Rebuilding ports that depend on Perl

Ok, the default Perl version has been updated… But this broke all the Perl ports! So it’s time to fix them by rebuilding against the newer Perl 5.28. Luckily the UPDATING notice points us to a simple way to do this:

# portmaster -f `pkg shlib -qR libperl.so.5.26`

And that’s it! Congratulations on updating an old test system via ports.

At last: All done!

What if something goes wrong?

You know your system and applications, are proficient with your shell and you’ve read UPDATING. What could possibly go wrong? Well, we’re dealing with computers here. Something really strange can go wrong anytime. It’s not very likely, but sometimes things happen.

Portmaster can help you if you ask for it before attempting upgrades. Before deinstalling an old package, it creates a backup. However after installing the new version it throws it away. But you can make it keep the backup by supplying the -b flag. Sometimes the old package can come in handy if something goes completely sideways. If you need backup packages, have a look in /usr/ports/packages/portmaster-backup. You can simple pkg add those if you need the old version back (of course you need to be sure that you didn’t update the packages dependencies or you need the downgrade them again, too!).

If you want to be extra cautious when updating that very special old box (that nobody dared to touch for nearly a decade because the boss threatened to call down terrible curses (not the library!) upon the one who breaks it), portmaster will also support you. Use the -w flag and have it preserve old shared libs before deinstalling an old package. I wouldn’t normally do it (and think my example made it clear that it’s really special). In fact I’ve never used it. But it might be good to know about it should you ever need it.

That said, on the more complicated boxes I usually create a temporary directory and issue a pkg create -a, completely backing up all the packages before I begin the update process. Usually I can throw away everything a while later, but having the backups saved me some pain a couple of times already.

In the end it boils down to: Letting your colleagues call you a coward or being the tough guy but maybe ruining your evening / weekend. Your choice.

Anf if you need to know even more about the tool that we’ve been using over and over now, just man portmaster.

What’s next?

I haven’t decided on the next topic that I’m going to write about. However I’ve planned for two more articles that will cover building from ports the modern way(tm) – and I hope that it will not take me another two years before I come to it…

Using FreeBSD with Ports (1/2): Classic way with tools

Installing software on Unix-like systems originally meant compiling everything from source and then copying the resulting program to the desired location. This is much more complicated than people unfamiliar with the subject may think. I’ve written an article about the history of *nix package management for anyone interested to get an idea of what it once was like and how we got what we have today.

FreeBSD pioneered the Ports framework which quickly spread across the other BSD operating systems which adopted it to their specific needs. Today we have FreeBSD Ports, OpenBSD Ports and NetBSD’s portable Pkgsrc. DragonflyBSD currently still uses DPorts (which is meant to be replaced by Ravenports in the future) and there are a few others like e.g. mports for MidnightBSD. Linux distributions have developed a whole lot of packaging systems, some closer to ports (like Gentoo’s Portage where even the name gives you an idea where it comes from), most less close. Ports however are regarded the classical *BSD way of bringing software to your system.

While the ports frameworks have diverged quite a bit over time, they all share a basic purpose:They are used to build binary packages that can then be installed, deinstalled and so on using a package manager. If you’re new to package management with FreeBSD’s pkg(8), I’ve written two introduction articles about pkg basics and pkg common usage that might be helpful for you.

Why using Ports?

OpenBSD’s ports for example are maintained only to build packages from and the regular user is advised to stay away from it and just install pre-built packages. On FreeBSD this is not the case. Using Ports to install software on your machines is nothing uncommon. You will read or hear pretty often that in general people recommend against using ports. They point to the fact that it’s more convenient to install packages, that it’s so much quicker to do so and that especially updating is much easier. These are all valid points and there are more. You’re now waiting for me to make a claim that they are wrong after all, aren’t you? No they are right. Use packages whenever possible.

But when it’s not possible to use the packages that are provided by the FreeBSD project? Well, then it’s best to… still use packages! That statement doesn’t make sense? Trust me, it does. We’ll come back to it. But to give you more of an idea of the advantages and disadvantages of managing a FreeBSD system with ports, let’s just go ahead and do it anyway, shall we?

Just don’t do this to a production system but get a freshly installed system to play with (probably in a VM if you don’t have the right hardware lying around). It still is very much possible to manage your systems with ports. In fact at work I have several dozen of legacy systems some of which begun their career with something like FreeBSD 5.x and have been updated since then. Of course ports were being used to install software on them. Unfortunately machines that old have a tendency to grow very special quirks that make it not feasible to really convert them to something easier to maintain. Anyways… Be glad if you don’t have to mess with it. But it’s still a good thing to know how you’d do it.

So – why would you use Ports rather than packages? In short: When you need something that packages cannot offer you. Probably you need a feature compiled in that is not activated by default and thus not available in FreeBSD’s packages. Perhaps you totally need the latest version in ports and cannot wait for the next package run to complete (which can take a couple of days). Maybe you require software licensed in a way that binary redistribution is prohibited. Or you’ve got that 9-STABLE box over there which you know quite well you shouldn’t run anymore. But because you relay on special hardware that offers drivers only for FreeBSD 9.x… Yeah, there’s always things like that and sometimes it’s hard to argue away. But when it comes to packages – since FreeBSD 9 is EOL for quite some time there are no packages being built for it anymore. You’ll come up with any number of other reasons why you can’t use packages, I’m not having any doubts.

Using Ports

Alright, so let’s get to it. You should have a basic understanding of the ports system already. If you haven’t, then I’d like to point you to another two articles written as an introduction to ports: Ports basics and building from ports.

If you just read the older articles, let me point out two things that have happened since then. In FreeBSD 11.0 a handy new option was added to portsnap. If you do

# portsnap auto

it will automatically figure out if it needs to fetch and extract the ports tree (if it’s not there) of if fetching the latest changesets and updating is the right thing to do.

The other thing is a pretty big change that happened to the ports tree in December 2017 not too long after I wrote the articles. The ports framework gained support for flavors and flavored ports. This is used heavily for Python ports which can often build for Python 2.7, 3.5, 3.6, … Now there’s generally only one Port for both Python2 and Python3 which can build the py27-foo, py36-foo, py37-foo and so on packages. I originally intended to cover tool-assisted ports management shortly after the last article on Ports, but after that major change it wasn’t clear if the old tools would be updated to cope with flavors at all. They were eventually, but since then I never found the time to revisit this topic.

Scenario

I’ve setup a fresh FreeBSD 11.2 system on my test machine and selected to install the ports that came with this old version. Yes, I deliberately chose that version, because in a second step we’re going to update the system to FreeBSD 11.3.

Using two releases for this has the advantage that it’s two fixed points in time that are easy to follow along if you want to. The ports tree changes all the time thanks to the heroic efforts that the porters put into it. Therefor it wouldn’t make sense to just use the latest version available because I cannot know what will happen in the future when you, the reader, might want to try it out. Also I wanted to make sure that some specific changes happened in between the two versions of the ports tree so that I can show you something important to watch out for.

Mastering ports – with Portmaster

There are two utilities that are commonly used to manage ports: portupgrade and portmaster. They pretty much do the same thing and you can choose either. I’m going to use portmaster here that I prefer for the simple reason of being more light-weight (portupdate brings in Ruby which I try to avoid if I don’t need it for something else). But there’s nothing wrong with portupgrade otherwise.

Building portmaster

You can of course install portmaster as a package – or you can build it from ports. Since this post is about working with ports, we’ll do the latter. Remember how to locate ports if you don’t know their origin? Also you can have make mess with a port without changing your shell’s pwd to the right directory:

# whereis portmaster
portmaster: /usr/ports/ports-mgmt/portmaster
# make -C /usr/ports/ports-mgmt/portmaster install clean

Portmaster build options

As soon as pkg and dialog4ports are in place, the build options menu is displayed. Portmaster can be built with completions for bash and zsh, both are installed by default. Once portmaster is installed and available, we can use it to build ports instead of doing so manually.

Building / installing X.org with portmaster

Let’s say we want to install a minimal X11 environment on this machine. How can we do it? Actually as simple as telling portmaster the origin of the port to install:

# portmaster x11/xorg-minimal

Port options…

After running that command, you’ll find yourself in a familiar situation: Lots and lots of configuration menus for port options will be presented, one after another. Portmaster will recursively let you configure the port and all of its dependencies. In our case you’ll have 42 ports to configure before you can go on.

Portmaster in action

In the background portmaster is already pretty busy. It’s gathering information about dependencies and dependencies of dependencies. It starts fetching all the missing distfiles so that the compilation can start right away when it’s the particular port’s turn. All the nice little things that make building from ports a bit faster and a little less of a hassle.

Portmaster: Summary for building xorg-minimal (1/2)

Once it has all the required information gathered and sorted out, portmaster will report to you what it is about to do. In our case it wants to build and install 152 ports to fulfill the task we gave it.

Portmaster: Summary for building xorg-minimal (2/2)

Of course it’s a good idea to take a look at the list before you acknowledge portmaster’s plan.

Portmaster: distfile deletion?

Portmaster will ask whether to keep the distfiles after building. This is ok if you’re building something small, but when you give it a bigger task, you probably don’t want to sit and wait to decide on distfiles before the next port gets built… So let’s cancel the job right there by hitting Ctrl-C.

Portmaster: Ctrl-C report (1/2)

When you cancel portmaster, it will try to exit cleanly and will also give a report. From that report you can see what has already been done and what still needs to be done (in form of a command that would basically resume building the leftovers).

Portmaster: Ctrl-C report (2/2)

It’s also good to know that there’s the /tmp/portmasterfail.txt file that contains the command should you need it later (e.g. after you closed your terminal). Let’s start the building process again, but this time specify the -D argument. This tells portmaster to keep the distfiles. You could also specify -d to delete them if you need the space. Having told portmaster about your decision helps to not interrupt the building all the time.

# portmaster -D x11/xorg-minimal

Portmaster: Build error

All right, there we have an error! Things like this happen when working with older ports trees. This one is a somewhat special case. Texinfo depends on one file being fetched that changes over time but doesn’t have any version info or such. So portmaster would fetch the current file, but that has silently changed since the distfile information was put into our old ports tree. Fortunately it’s nothing too important and we can simply “fix” this by overwriting the file’s checksum with the one for the newer file:

# make makesum -C /usr/ports/print/texinfo
# portmaster -D x11/xorg-minimal

Manually fetching missing distfile

Next problem: This old version of mesa is no longer available in the places that it used to be. This can be fixed rather easily if you can find the distfile somewhere else on the net! Just put it into its place manually. Usually this is /usr/ports/distfiles, but there are ports that use subdirectories like /usr/ports/distfiles/bash or even deeper structures like e.g. /usr/ports/distfiles/xorg/xserver!

# fetch https://mesa.freedesktop.org/archive/older-versions/17.x/mesa-17.3.9.tar.xz -o /usr/ports/distfiles/mesa-17.3.9.tar.xz
# portmaster -D x11/xorg-minimal

Portmaster: Success!

Eventually the process should end successfully. Portmaster gives a final report – and our requested program has been installed!

What’s next?

Now you know how to build ports using portmaster. There’s more to that tool, though. But we’ll come to that in the next post which will also show how to use it to update ports.

FreeBSD: Building software from ports (2/2)

My previous post discussed what ports are, where they can be found on FreeBSD and what the files of which a port is composed of look like. This post will now detail how to use ports to build software on FreeBSD (the other BSDs have ports trees that work somewhat similar but are not identical. There are important differences!).

Packages and ports: A word of warning

The ports system works hand in hand with FreeBSD’s package manager Pkg. It makes little difference if some software on your machine was installed via a package or directly from ports – packages are in fact actually built from ports! Still it is not really recommended to mix packages and ports. In past times it was strongly discouraged. Things have changed since then. I’ve done it a lot – and mostly got away with it. Don’t rely on it, though, especially if you’re new to the whole topic. Feel free to do it on a test system and be completely happy – or face subtle and annoying breakage. You cannot know up front.

What’s the deal here? Modern software is a complex thing. Most programs rely on other programs or external libraries. A lot of programs can be configured at run time in certain ways. There are however decisions about program functionality that have to be made at compile time. The ports system allows you to build software with compile-time options other than the default. Pre-compiled packages have no chance to know that you choose to deactivate an option when you built a library yourself that they make use of. They assume that this feature is present (it was available on the system the package was built on after all!). And what can one poor program do in that case? Crash, explode, malfunction… A lot of things.

And then there’s the problem of mixing versions which can lead to all kinds of fun. If you stick with either ports or packages, you always have a consistent system with versions that are known to play together well (as long as the maintainers do their job well – we’re all humans and errors do occur).

Just keep that in mind when thinking about mixing programs installed from packages and ports on one system. You can do that. But it doesn’t mean you should. Enabling more options is generally safer than removing ones set by default. It can still have consequences. This is Unix though. Do whatever you see fit – and claim the responsibility. Your choice.

Most basic ports building

Building a software from ports is extremely easy. Go to the directory of a port and type make. Yes, that’s all! Let’s assume the port has no unsatisfied dependencies. The ports system will then check to see if the source code tarball is present in /usr/ports/distfiles. If it isn’t, it will automatically download it. Then it extracts the source code, prepares everything for the compilation and compiles it.

Building the ‘pkg’ port

On my fresh example system I build the Pkg manager from ports first – it’s needed for every other port anyway. Once everything has finished I get my shell back.

Building of Pkg completed

Installing the program is just as easy: Use make install

Installing the newly built port

That’s it, Pkg is now installed. We’re basically done with that port. However there’s still the “work” directory left over from the building process. To tidy up our port’s directory we can issue make clean.

Cleaning up after the build

Dependency handling

On to a just slightly more complex example. I want to build and install an old version of the LUA interpreter which depends on another port, libedit. Of course I could build devel/libedit first and then lang/lua51. In that case it wouldn’t be so bad. But if you think of larger programs with hundreds of dependencies that approach would be a nightmare.

So what to do about it? Well, nothing actually. The ports system takes care of it automatically! Just have it build LUA and it will figure out that it has to build the dependency first.

Building, installing and cleaning up in one command

The parameters to make that we used above are called make targets, BTW, and can be combined. That means it’s perfectly fine to issue make install clean together as you can see in the picture above.

Dependencies are handled automatically

The clean make target is also applied to all ports that were built as a dependency for the current port. Things like this make ports very convenient to use.

More on make and targets

Make targets can depend on other make targets. When you issue make install these are the targets that are actually run:

  • make config (more on that in a minute)
  • make fetch (fetch all files needed to build the port)
  • make checksum (check integrity of downloaded file(s))
  • make depends (check for missing dependencies and build/install those)
  • make extract (extract distfile(s) for the port)
  • make patch (apply patches for this port, if any)
  • make build (actually build the port)
  • make install (install the newly-built program)

If you type make checksum for example, all targets up to and including that one will run (that is config, fetch and checksum in that case). Running just make without any target will assume the default target which is equivalent to make build.

Also make will take an argument to look for the Makefile in another directory if you wish. So instead of doing e.g. this:

# cd /usr/ports/archivers/bzip2
# make install clean

you could also simply do this:

# make -C /usr/ports/archivers/bzip2 install clean

You’re in control: Ports options

So far it’s all nice and well but there’s no real advantage to using ports instead of packages. May I introduce ports options? Let’s say you we want to build BASH. If issue make in shells/bash, this is what happens:

Build options for BASH

The port ports-mgmt/dialog4ports is fetched and installed. It’s so small that you might miss it but it’s quite important. It’s needed to display the menu in the picture above which lets you set various options for the port.

You can now e.g. choose to not install the documentation if you’re short on space on a small or embedded system (sure, you wouldn’t actually compile on such a system, but that’s only an example, right?). If you don’t want BASH to support any foreign languages, deselect NLS. In case you feel that BASH’s built-in help is useless (did you ever issue the help command when you ran BASH?), you can cut that feature. Things like that.

If you see the option configuration for a port the first time, you see the default configuration. In general it’s a good idea to leave options alone if you’re in doubt what they do (do a little research if you have the time). Of course you’re also free to experiment with them. It’s your system.

Once you’re happy, accept your selection and the source tarball is being fetched, extracted, etc. You know the score.

Build options for bison

But what’s that? Another configuration menu (for bison)? And another (m4) and another (texinfo), etc… It’s 8 menus for a rather basic program like BASH! And worse: The building process will run and build dependencies and when a port with options is reached, the process is interrupted and prompts the user.

Now imagine you’re building a whole graphical desktop like MATE… Currently even the basic desktop would build no less that 338 dependency packages on a fresh system! And there’s quite a few ports on the list which build rather heavy software that takes it’s time compiling. It would totally make sense to let it build over night or at least not require you to keep staring at the screen, waiting for the next options selection to confirm, right?

Recursive operations

Exactly that’s why recursive operations are supported by the ports system. The standard make target that was implicitly run to open the options dialog is make config. The recursive option which would run the same on each and every port that’s listed as a dependency for the current port is make config-recursive.

If you want to build MATE as mentioned in the previous example, that would start a true marathon of options for you to configure. However it’s still a lot better to be able to do this up front so that the build process can run uninterruptedly afterwards.

Oh, and don’t be surprised if you went through it all only to find that still another configuration dialog pops up later! Why? Most likely you enabled an option on some package that made it depend on another package that’s not a dependency by default. And that package may need to have its options configured, too. So if you changed any options it makes sense to run make config-recursive again until no more new option dialog windows are displayed!

Recursively fetching distfiles for security/sudo

You can also do make fetch-recursive to fetch the distfiles for the current port and all dependencies. Again: Keep in mind that enabling more options may lead to new dependencies. If you want to make sure that you have all the distfiles, you might want to run make fetch-recursive again after changing ports options.

Other things to know

Wonder where the all the options are saved? They are stored in text files in /var/db/ports/category_portname. But there’s no need to edit or delete them; if you want to get rid of them, there’s make rmconfig to do that. Also make rmconfig-recursive exists if you feel like blowing away a huge amount of them.

Ports options in /var/db/ports

Another thing that comes in handy is make build-depends-list which will show you a list of ports that will be built as build dependencies for your current port. If you want to see the runtime dependencies you would use make run-depends-list. And then there’s also make all-depends-list which will show you each and every port that would be installed if you chose to build the current port.

Showing port dependencies

You should also know that you can deinstall a port by using make deinstall. Yes, it is also possible to remove the package using pkg delete but that will lead to a problem. The ports infrastructure keeps track of installed ports separately and Pkg does not know anything about this. So even if your package is removed, the Ports infrastructure will insist that it is still installed and there’s something very wrong with your system!

Now what to do if you have that case? Use make reinstall to install the package again even though ports thinks that it’s already installed.

More on ports?

To be honest, there’s quite a bit more to ports than I could cover here. You may want to man 7 ports to see what other targets are available and what they do. Also we haven’t even touched how to keep your system updated when using ports!

The ports infrastructure is a great means of installing customized programs on your system. It’s quite easy to use as you’ve seen. But things can be made even easier – which is why there are helper tools available. I will write a follow-up article covering those (not the next one, though). But for now enjoy all of those new possibilities with software on your FreeBSD machines!

FreeBSD: Building software from ports (1/2)

In my previous two (link) posts (link) I wrote about using Pkg, FreeBSD’s package manager.

Pre-built binary packages are convenient to use but sometimes you need some more flexibility, want an application that cannot not be distributed in binary form due to license issues (or have some other requirements). Building software by hand is certainly possible – but with all the things involved, this can be a rather tedious process. It’s also slow, error-prone and there’s often no clean way to get rid of that stuff again. FreeBSD Ports to the rescue!

This first part is meant as a soft introduction to FreeBSD’s ports, assuming no prior knowledge (if I fail to explain something, feel free to comment on this post). It will give you enough background information to understand ports enough to start using them in the next article.

What “Ports” are

When programmers talk about porting something over, what they originally meant is this: Take an application that was written with one processor architecture in mind (say i386) and modify the source so that it runs on another (arm64 for example) afterwards. The term “porting” is also used when modifying the source of any program to make it run on another OS. The version that runs on the other architecture/OS is called a port of the original program to a different platform.

FreeBSD uses the term slightly differently. There’s a lot of software written e.g. for Linux that will build and work on FreeBSD just fine as it is. Even though it does not require any changes, that software might be ported to FreeBSD. So in this case “porting” does not mean “make it work at all” but make it easily available. This is done by creating a port for any program. That term doesn’t mean a variant of the source code in this case but rather a means to give you easy access to that software on FreeBSD.

So what is a port in FreeBSD? Actually a port is a directory with a bunch of files in it. The heart of it is one file that basically is a recipe if you will. That recipe contains everything needed to build and install the port (and thus have the application installed on your machine in the end). Following this metaphor you could think of all the ports as a big cookbook. Formally it is known as the Ports collection. All those files in your filesystem related to ports are refered to as the Ports tree.

How to get the Ports tree

There are several options to obtain a copy of the ports tree. When you install FreeBSD you can decide whether or not to install it, too. I usually don’t do that because on systems that use binary packages only. It wastes only about 300 MB of space, but more importantly consists of almost 170.000 files (watch your inodes on embedded devices!). Take a look at /usr/ports: If that directory is empty your system is currently missing the ports tree.

The simplest way to get it is by using portsnap:

# portsnap fetch extract

If you want to update the tree later, you can use:

# portsnap fetch update

Another way is to use Subversion. This is more flexible: With portsnap you always get the current tree while Subversion also allows you to checkout older revisions, too. If you plan to become a ports developer, you will probably want to use Subversion for tools like svndiff. If you just want to use ports, portsnap should actually suffice. All currently supported versions of FreeBSD contain a light-weight version of Subversion called svnlite.

Here’s how to checkout the latest tree:

# svnlite checkout https://svn.freebsd.org/ports/head /usr/ports

If you want to update it later run:

# svnlite update /usr/ports

Old versions of the tree

You normally shouldn’t need these but it’s good to know that they exist. Using Subversion you can also retrieve old trees. Be sure that /usr/ports is empty (including for Subversion’s dot directories) or Subversion will see that there’s already something there and won’t do the checkout. If for example you want the ports tree as it existed in 2016Q4, you can retrieve it like this:

# svnlite checkout https://svn.freebsd.org/ports/branches/2016Q4 /usr/ports

There are also several tags available that allow to get certain trees. Maybe you want to see which ports were available when FreeBSD 9.2 was released. Get the tree like this:

# svnlite checkout https://svn.freebsd.org/ports/tags/RELEASE_9_2_0 /usr/ports

And if you need the last tree that is guaranteed to work with 9.x there’s another special tag for it:

# svnlite checkout https://svn.freebsd.org/ports/tags/RELEASE_9_EOL /usr/ports

Keep in mind though that using old trees is risky because they contain program versions with vulnerabilities that have since been found! Also mind that it’s NOT a smart thing to simply get the tree for RELEASE_7_EOL because it still holds a port for PHP 5.2 and you thought that it would be cool to offer your customers as many versions as possible. Yes, it may be possible that you can still build it if you invest some manual work. But no, that doesn’t make it a good idea at all.

Oh, and don’t assume that old ports trees will be of any use on modern versions of FreeBSD! The ports architecture changed quite a bit over time, the most notable change being the replacement of the old pkg_* tools with the new Pkg. Ports older than a certain time definitely won’t build in their old, unmodified state today (and I say it again: You really shouldn’t bother unless you have a very special case).

Port organization

Take a look at the contents of /usr/ports on a system that has the tree installed. You will find over 60 directories there. There are a few special ones like distfiles (where tarballs with program’s source code get stored – might be missing initially) or Mk that holds include files for the ports infrastructure. The others are categories.

If you’re looking for a port for Firefox, that will be in www. GIMP is in graphics and it’s probably no surprise that Audacious (a music player) can be found in audio. Some program’s categories will be less obvious. LibreOffice is in editors which is not so bad. But help2man for example is in misc and not in converters or devel as at least I would expect if I didn’t know. In general however after a while of working with ports you will have a pretty good chance to guess where things are.

Say we are interested in the port for the window manager Sawfish for example. It’s located in /usr/ports/x11-wm/sawfish. Let’s take a closer look at that location and take it apart:

/usr/ports is the “ports directory”.
x11-wm (short for X11 window managers) is the category.
sawfish is the individual port’s name.

When referring to where a port lives, you can omit the ports directory since everybody is assumed to know where it is. The important information when identifying a port is the category and the name. Together those form what is known as the port origin (x11-wm/sawfish in our case).

How to find a port in the tree

There are multiple methods to find out the origin for the port you are looking for. Probably the simplest one is using whereis. If we didn’t know that sawfish is in x11-wm/sawfish we could do this:

% whereis sawfish
sawfish: /usr/ports/x11-wm/sawfish

This does however only work if you know the exact name of the port. And there’s a little more to it: Sometimes the name of a port and a package differ! This is often the case for Python-based packages. I have SaltStack installed, for example. It’s a package called py27-salt:

% pkg info -x salt
py27-salt-2017.7.1_1

If we were to look for that, we wouldn’t find it:

% whereis py27-salt
py27-salt:

So where is the port for the package?

% pkg info py27-salt
py27-salt-2017.7.1_1
Name           : py27-salt
[...]
Origin         : sysutils/py-salt
[...]

Here you can see that the port’s name is py-salt! The “27” gets added when the package is created and reflects the version of Python that it’s build against. You may also see some py3-xyz ports. In those cases the name reflects that the port cannot be built with Python 2.x. The package will still be called py36-xyz, though (or whatever the default Python 3.x version is at that time)!

When discussing package management I recommended FreshPorts and when working with ports it can be useful, too. Search for some program’s name and it might be easier for you to find the package name and the port origin for it!

What a port looks like

Let’s take a look at the port for the zstd compression utility:

% ls /usr/ports/archivers/zstd/
distinfo	Makefile	pkg-descr	pkg-plist

So what have we here? The simplest file is pkg-descr. Each package has a short and a long package description – this file is what contains the latter: A detailed description that should give you a good idea whether this port would satisfy your needs:

% cat /usr/ports/archivers/zstd/pkg-descr
Zstd, short for Zstandard, is a real-time compression algorithm providing
high compression ratios.  It offers a very wide range of compression vs.
speed trade-offs while being backed by a very fast decoder.  It offers
[...]

Then there’s a file called distinfo. It lists all files that need to be downloaded to build the port (usually the program’s source code). It also contains a checksum and the file’s size to make sure that the valid file is being used (an archive could get corrupted during the transfer or you could even get an archive that somebody tempered with!):

% cat /usr/ports/archivers/zstd/distinfo 
TIMESTAMP = 1503324578
SHA256 (facebook-zstd-v1.3.1_GH0.tar.gz) = 312fb9dc75668addbc9c8f33c7fa198b0fc965c576386b8451397e06256eadc6
SIZE (facebook-zstd-v1.3.1_GH0.tar.gz) = 1513767

There’s usually also pkg-plist. It lists all the files that are installed by the port:

% cat /usr/ports/archivers/zstd/pkg-plist 
bin/unzstd
bin/zstd
bin/zstdcat
[...]
lib/libzstd.so.%%PORTVERSION%%
libdata/pkgconfig/libzstd.pc
man/man1/unzstd.1.gz
man/man1/zstd.1.gz
man/man1/zstdcat.1.gz

And finally there’s the Makefile. This is where all the magic happens. If you’re a programmer or you have built software from source before, there’s a high chance that you’re at least somewhat familiar with a tool called make. It processes Makefiles and then does as told by those. While it’s most often used to compile software it can actually be used for a wide variety of tasks.

If you don’t have at least some experience with them, Makefiles look pretty much obscure and creating them seems like a black art. If you’ve ever looked at a complicated Makefile, you may be worried to hear that to use ports you have to use make. Don’t be. The people who take care of the Ports infrastructure are the ones who really need to know how to deal with all the nuts and bolts of make. They’ve already solved all the common tasks so that the porters (those people who create the actual ports) can rely on it. This is done by including other Makefiles and it manages to hide away all the scariness. And for you as a user things are even simpler as you can just use what others created for you!

Let’s take a look at the Makefile for our example port:

% cat /usr/ports/archivers/zstd/Makefile 
# Created by: John Marino <marino@FreeBSD.org>
# $FreeBSD: head/archivers/zstd/Makefile 448492 2017-08-21 20:44:02Z sunpoet $

PORTNAME=	zstd
PORTVERSION=	1.3.1
DISTVERSIONPREFIX=	v
CATEGORIES=	archivers

MAINTAINER=	sunpoet@FreeBSD.org
COMMENT=	Zstandard - Fast real-time compression algorithm

LICENSE=	BSD3CLAUSE GPLv2
[...]
post-patch:
	@${REINPLACE_CMD} -e 's|INSTALL_|BSD_&|' ${WRKSRC}/lib/Makefile ${WRKSRC}/programs/Makefile

.include <bsd.port.mk>

Now that doesn’t look half bad for a Makefile, does it? In fact it’s mostly just defining Variables! The only line that looks somewhat complex is the “post-patch” command (which is also less terrifying than it first looks – if you know sed you can surely guess what it’ll do).

There can actually be more files in some ports. If FreeBSD-specific patches are required to build the port, those are included in the ports tree. You can find them in a sub-directory called files located in the port’s directory. Here’s an example:

% ls /usr/ports/editors/vim/files/
patch-src-auto-configure        vietnamese_viscii.vim
patch-src-installml.sh          vimrc

The patches there are named after the files that they apply to. Every patch in the files directory is automatically applied when building the port.

What’s next?

Alright. With that we’ve got a basic overview of what Ports are covered. The next post will show how to actually use them to build and install software.