Using FreeBSD with Ports (2/2): Tool-assisted updating

In the previous post I explained why sometimes building your software from ports may make sense on FreeBSD. I also introduced the reader to the old-fashioned way of using tools to make working with ports a bit more convenient.

In this follow-up post we’re going to take a closer look at portmaster and see how it especially makes updating from ports much, much easier. For people coming here without having read the previous article: What I describe here is not what every FreeBSD admin today should consider good practice (any more)! It can still be useful in special cases, but my main intention is to discuss this for building up the foundation for what you actually should do today.

Building a desktop

Last time we stopped after installing the xorg-minimal meta-port. Let’s say we want a very simple desktop installed on this machine. I chose the most frugal *nix desktop, EDE (the Equinox Desktop Environment, looking kind of like Win95), because that’s drawing in things that I need for demonstrating a few interesting things here – and not that much more.

Unfortunately in the ports tree that we’re using, exactly that port is broken (the newer compiler in FreeBSD 11.2 is more picky than the older ones and not quite happy with EDE code). So to go on with it, we have to fix it first. I’ve uploaded an additional patch file from a later version of the port and also prepared a patch for the port’s Makefile. If you want to follow along, you can just copy the three lines below to your terminal:

# fetch http://www.elderlinux.org/files/patch-evoke_Xsm.cpp -o /usr/ports/x11-wm/ede/files/patch-evoke_Xsm.cpp
# fetch http://www.elderlinux.org/files/patch_ede_port -o /usr/ports/x11-wm/ede/patch_ede_port
# patch -d /usr/ports/x11-wm/ede -i patch_ede_port

Using Portmaster to build and install EDE

Thanks to build-time dependencies and default options in FreeBSD it’s still another 110 ports to build, but that’s fine. We could remove some unneeded options and cut it down quite a bit. Just to give you an idea: By configuring only one package (doxygen) to not pull in all the dependencies that it usually does, it would be just 55 (!) ports.

But let’s say we’re lazy. Do we have to face all of those configure dialogs (72 in cause you are curious)? No, we don’t. That’s why portmaster has the -G flag which skips the config make target and just uses the standard port options:

# portmaster -DG x11-wm/ede

EDE was successfully installed

Using this option can be a huge time-saver if you’re building something where you know that you don’t need to change the options for the application and its dependencies.

System update

Now that we have a simple test system with 265 installed but outdated packages. Let’s update it! Remember that unlike e.g. Linux, FreeBSD keeps third party software installed from packages or ports and the actual operating system separate. We’ll update the latter first:

# freebsd-update upgrade -r 11.3-RELEASE

With this command, we make the updater download the required files for the upgrade from 11.2-RELEASE to 11.3-RELEASE.

Upgrading FreeBSD to 11.3-RELEASE

When it’s done, and we’ve closed the three lists with the removed, updated and new files, we can install the new kernel:

# freebsd-update install

Once that’s done, it’s time to reboot the system so the new kernel boots up. Then the userland part of the operating system is installed by using the same command again:

# shutdown -r now
# freebsd-update install

Kernel upgrade complete

Preparations

Now in our fresh 11.3 system we should first get rid of the old ports tree to replace it with a newer one, right? Wait, hold that rm command for a second and let me show you something really useful!

If you take a look at the /usr/ports directory, you’ll find a file appropriately named UPDATING. And since that’s right what we were about to do, why not take a look at it?

So what is this? It’s an important file for people updating their systems using ports. Here is where ports maintainers document breaking changes. You are free to ignore it and the advice that it gives and there’s actually a chance that you’ll get away with it and everything will be fine. But sometimes you don’t – and fixing stuff that you screwed up might take you quite a bit longer than at least skimming over UPDATING.

# less /usr/ports/UPDATING

But right now it’s completely sufficient to look at the metadata of the first notification which reads:

20180330:
  AFFECTS: users of lang/perl5*
  AUTHOR: mat@FreeBSD.org

The main takeaway here is the date. The last heads-up notice for our old ports tree was on 2018-03-30.

Checking out a newer ports tree

Now let’s throw it all away and then get the new ports tree. Usually I’d use portsnap for this, but in this case I want a special ports tree (the one that would have come with the OS if I got ports from a fresh 11.3 installation), so I’m checking it out from SVN:

# rm -rf /usr/ports/.* /usr/ports/*
# svnlite co svn://svn.freebsd.org/ports/tags/RELEASE_11_3_0 /usr/ports

If you’re serious about updating a production server that you care about, now is the time to read through UPDATING again. Search for the date string that you previously took a note of and then read the messages all the way up to the beginning of the file. It’s enough to read the AFFECTS lines until you hit one message that describes a port which you are using. You can ignore all the rest but should really read those heads-up messages that affect your system.

What software can be updated?

BTW… You know which packages you have installed, don’t you? A huge update like what we’re facing here takes some planning up-front if you want to do it in a professional manner. In general you should update much more often, of course! This makes things much, much easier.

Updating from ports

Ok, we’re all set. But which software can be updated? You can ask pkg(8) to compare installed packages to the respective distinfo from the corresponding port:

# pkg version -l "<"

If you pipe that into wc -l you will see that 165 of the 265 installed packages can (and probably should) be updated.

Updating software from ports

We’ll start with something really simple. Let’s say, we just want to update pkgconf for now. How do we do that with portmaster? Easy enough: Just like we would have portmaster install it in the first place. If something is already installed, the tool will figure out if an update is available and then offer to update it.

# portmaster -G devel/pkgconf

And what will happen if the port is already up to date? Well, then portmaster will simply re-build and re-install it.

Partial update finished

While partial updates are possible, it’s a much better idea to keep the whole system updated (if at all possible). To do that, you can use the -a flag for portmaster to update all installed software.

Another nice flag is -i, which is for interactive mode. In this mode, portmaster will ask for every port if it should be updated or not. If you’re leaving out critical ports (dependencies) this can lead to an impossible update plan and portmaster will not start updating. But it can be useful for cherry-picking.

Interactive update mode

Now let’s attempt to upgrade all the ports, shall we?

# portmaster -aG

As always, portmaster will show you its plan and ask for your confirmation. There are two things that probably deserve a word or two about them. Usually the plan is to update an application, but sometimes portmaster wants to re-install or even downgrade them (see picture)! What’s happening here?

Re-installs mostly happen when a dependency changed and portmaster figured out that it might be a good idea to rebuild the port against the newer version of the dependency, even though there is no newer version available for the actual application.

Upgrading, “downgrading”, re-installing

Downgrades are a different thing. They can happen when you installed something from a newer ports tree and then go back to an older one (something you usually shouldn’t do). But in this case it’s actually a false claim. Portmaster will not downgrade the package – it was merely confused by the fact that the versioning scheme changed (because of the 0 in 2018.4 it thinks that this version is older than the previous 2.1.3…).

Moved ports

If you’re paying close attention to all the information that portmaster gives you, you’ll have seen lines like the following one:

===>>> The x11/bigreqsproto port moved to x11/xorgproto

There’s another interesting file in the ports tree called MOVED. It keeps track of moved or removed ports. Sometimes ports are renamed or moved to another category if the maintainer decides it fits better. Portmaster for example started as sysutils/portmaster and was later moved when the ports-mgmt category was introduced. However you won’t find this information in the MOVED file – because it happened before the time that the current MOVED keeps records for (i.e. early 2007 in this case).

The example above is due to the fact that last year the upstream project (Xorg) decided to combine the protocol headers into one distribution package. Before that there were more than 20 separate packages for them (and before that, once upon a time, all of Xorg had been one giant monolithic release – but I digress…)

Problem with merged ports

The good news here is that portmaster is smart enough to parse the MOVED file and figure out how to deal with this kind of changes in the ports tree! The bad news is that this does not work for more complicated things like the merges that we just talked about…

So what now? Good thing you read the relevant UPDATING notification, eh?

20180731:
  AFFECTS: users of x11/xorg and all ports with USE_XORG=*proto
  AUTHOR: zeising@FreeBSD.org

Bulk-deleting obsolete ports and trying again

So let’s first get rid of the old *proto packages with the command that developer Niclas Zeising proposes and then try again:

# pkg version -l \? | cut -f 1 -w | grep -v compat | xargs pkg delete -fy
# portmaster -aG

Required options

Alright, we have one more problem to overcome. There are ports that will fail to build if we run portmaster with the -G flag. Why? Because they have mandatory ports options where you need to choose from things like a backend or a certain mechanic.

The “mandatory options” case

One such case is freetype2. Since this one fails, build it separately and do not skip the config dialog for this one:

# portmaster print/freetype2

Once that’s done, we can continue with updating all the remaining ports. After quite a while (because LLVM is a beast) all should be done!

Updating complete!

Default version changes

Did you read the following notice in UPDATING?

20181213:
  AFFECTS: users of lang/perl5*
  AUTHOR: mat@FreeBSD.org

For the big update run we ignored this. And in fact, portmaster did update Perl, but only to the latest version in the 5.26 branch of the language:

# pkg info -x perl
perl5-5.26.3

Why? Well, because that was the version of Perl that was already installed. Actually this is ok, if you’re not using Perl yourself and thus can live without the latest features added. However if you want to (or have to) upgrade to a later Perl major version we have a little bit more work to do.

First edit /etc/make.conf and put the following line in there:

DEFAULT_VERSIONS+=perl5=5.28

This is a hint to the ports framework that whenever you’re building a Perl port, you want to build it against this version. If you don’t do that, you will receive a warning when building the other Perl. Because in this case you’re installing an additional Perl version but all the ports will use the primary one. So more likely than not you don’t want to do this.

Upgrading Perl

Next we need to build the newer Perl. The special thing here is that we need to tell portmaster of the changed origin so that the new version actually replaces the old one. We can do this by using the -o flag. Mind the syntax here, it’s new origin first and then old origin and not vice versa (took me a while to get used to it…)!

But let’s check the origin real quick, before we go on. The pkg command above showed that the package is called perl5. This outputs what we wanted to know:

# pkg info perl5
perl5-5.26.3
Name           : perl5
Version        : 5.26.3
Installed on   : Tue Sep 10 21:15:36 2019 CEST
Origin         : lang/perl5.26
[...]

There we have it. Now portmaster can begin doing its thing:

# portmaster -oG lang/perl5.28 lang/perl5.26

Rebuilding ports that depend on Perl

Ok, the default Perl version has been updated… But this broke all the Perl ports! So it’s time to fix them by rebuilding against the newer Perl 5.28. Luckily the UPDATING notice points us to a simple way to do this:

# portmaster -f `pkg shlib -qR libperl.so.5.26`

And that’s it! Congratulations on updating an old test system via ports.

At last: All done!

What if something goes wrong?

You know your system and applications, are proficient with your shell and you’ve read UPDATING. What could possibly go wrong? Well, we’re dealing with computers here. Something really strange can go wrong anytime. It’s not very likely, but sometimes things happen.

Portmaster can help you if you ask for it before attempting upgrades. Before deinstalling an old package, it creates a backup. However after installing the new version it throws it away. But you can make it keep the backup by supplying the -b flag. Sometimes the old package can come in handy if something goes completely sideways. If you need backup packages, have a look in /usr/ports/packages/portmaster-backup. You can simple pkg add those if you need the old version back (of course you need to be sure that you didn’t update the packages dependencies or you need the downgrade them again, too!).

If you want to be extra cautious when updating that very special old box (that nobody dared to touch for nearly a decade because the boss threatened to call down terrible curses (not the library!) upon the one who breaks it), portmaster will also support you. Use the -w flag and have it preserve old shared libs before deinstalling an old package. I wouldn’t normally do it (and think my example made it clear that it’s really special). In fact I’ve never used it. But it might be good to know about it should you ever need it.

That said, on the more complicated boxes I usually create a temporary directory and issue a pkg create -a, completely backing up all the packages before I begin the update process. Usually I can throw away everything a while later, but having the backups saved me some pain a couple of times already.

In the end it boils down to: Letting your colleagues call you a coward or being the tough guy but maybe ruining your evening / weekend. Your choice.

Anf if you need to know even more about the tool that we’ve been using over and over now, just man portmaster.

What’s next?

I haven’t decided on the next topic that I’m going to write about. However I’ve planned for two more articles that will cover building from ports the modern way(tm) – and I hope that it will not take me another two years before I come to it…

Advertisements

Using FreeBSD with Ports (1/2): Classic way with tools

Installing software on Unix-like systems originally meant compiling everything from source and then copying the resulting program to the desired location. This is much more complicated than people unfamiliar with the subject may think. I’ve written an article about the history of *nix package management for anyone interested to get an idea of what it once was like and how we got what we have today.

FreeBSD pioneered the Ports framework which quickly spread across the other BSD operating systems which adopted it to their specific needs. Today we have FreeBSD Ports, OpenBSD Ports and NetBSD’s portable Pkgsrc. DragonflyBSD currently still uses DPorts (which is meant to be replaced by Ravenports in the future) and there are a few others like e.g. mports for MidnightBSD. Linux distributions have developed a whole lot of packaging systems, some closer to ports (like Gentoo’s Portage where even the name gives you an idea where it comes from), most less close. Ports however are regarded the classical *BSD way of bringing software to your system.

While the ports frameworks have diverged quite a bit over time, they all share a basic purpose:They are used to build binary packages that can then be installed, deinstalled and so on using a package manager. If you’re new to package management with FreeBSD’s pkg(8), I’ve written two introduction articles about pkg basics and pkg common usage that might be helpful for you.

Why using Ports?

OpenBSD’s ports for example are maintained only to build packages from and the regular user is advised to stay away from it and just install pre-built packages. On FreeBSD this is not the case. Using Ports to install software on your machines is nothing uncommon. You will read or hear pretty often that in general people recommend against using ports. They point to the fact that it’s more convenient to install packages, that it’s so much quicker to do so and that especially updating is much easier. These are all valid points and there are more. You’re now waiting for me to make a claim that they are wrong after all, aren’t you? No they are right. Use packages whenever possible.

But when it’s not possible to use the packages that are provided by the FreeBSD project? Well, then it’s best to… still use packages! That statement doesn’t make sense? Trust me, it does. We’ll come back to it. But to give you more of an idea of the advantages and disadvantages of managing a FreeBSD system with ports, let’s just go ahead and do it anyway, shall we?

Just don’t do this to a production system but get a freshly installed system to play with (probably in a VM if you don’t have the right hardware lying around). It still is very much possible to manage your systems with ports. In fact at work I have several dozen of legacy systems some of which begun their career with something like FreeBSD 5.x and have been updated since then. Of course ports were being used to install software on them. Unfortunately machines that old have a tendency to grow very special quirks that make it not feasible to really convert them to something easier to maintain. Anyways… Be glad if you don’t have to mess with it. But it’s still a good thing to know how you’d do it.

So – why would you use Ports rather than packages? In short: When you need something that packages cannot offer you. Probably you need a feature compiled in that is not activated by default and thus not available in FreeBSD’s packages. Perhaps you totally need the latest version in ports and cannot wait for the next package run to complete (which can take a couple of days). Maybe you require software licensed in a way that binary redistribution is prohibited. Or you’ve got that 9-STABLE box over there which you know quite well you shouldn’t run anymore. But because you relay on special hardware that offers drivers only for FreeBSD 9.x… Yeah, there’s always things like that and sometimes it’s hard to argue away. But when it comes to packages – since FreeBSD 9 is EOL for quite some time there are no packages being built for it anymore. You’ll come up with any number of other reasons why you can’t use packages, I’m not having any doubts.

Using Ports

Alright, so let’s get to it. You should have a basic understanding of the ports system already. If you haven’t, then I’d like to point you to another two articles written as an introduction to ports: Ports basics and building from ports.

If you just read the older articles, let me point out two things that have happened since then. In FreeBSD 11.0 a handy new option was added to portsnap. If you do

# portsnap auto

it will automatically figure out if it needs to fetch and extract the ports tree (if it’s not there) of if fetching the latest changesets and updating is the right thing to do.

The other thing is a pretty big change that happened to the ports tree in December 2017 not too long after I wrote the articles. The ports framework gained support for flavors and flavored ports. This is used heavily for Python ports which can often build for Python 2.7, 3.5, 3.6, … Now there’s generally only one Port for both Python2 and Python3 which can build the py27-foo, py36-foo, py37-foo and so on packages. I originally intended to cover tool-assisted ports management shortly after the last article on Ports, but after that major change it wasn’t clear if the old tools would be updated to cope with flavors at all. They were eventually, but since then I never found the time to revisit this topic.

Scenario

I’ve setup a fresh FreeBSD 11.2 system on my test machine and selected to install the ports that came with this old version. Yes, I deliberately chose that version, because in a second step we’re going to update the system to FreeBSD 11.3.

Using two releases for this has the advantage that it’s two fixed points in time that are easy to follow along if you want to. The ports tree changes all the time thanks to the heroic efforts that the porters put into it. Therefor it wouldn’t make sense to just use the latest version available because I cannot know what will happen in the future when you, the reader, might want to try it out. Also I wanted to make sure that some specific changes happened in between the two versions of the ports tree so that I can show you something important to watch out for.

Mastering ports – with Portmaster

There are two utilities that are commonly used to manage ports: portupgrade and portmaster. They pretty much do the same thing and you can choose either. I’m going to use portmaster here that I prefer for the simple reason of being more light-weight (portupdate brings in Ruby which I try to avoid if I don’t need it for something else). But there’s nothing wrong with portupgrade otherwise.

Building portmaster

You can of course install portmaster as a package – or you can build it from ports. Since this post is about working with ports, we’ll do the latter. Remember how to locate ports if you don’t know their origin? Also you can have make mess with a port without changing your shell’s pwd to the right directory:

# whereis portmaster
portmaster: /usr/ports/ports-mgmt/portmaster
# make -C /usr/ports/ports-mgmt/portmaster install clean

Portmaster build options

As soon as pkg and dialog4ports are in place, the build options menu is displayed. Portmaster can be built with completions for bash and zsh, both are installed by default. Once portmaster is installed and available, we can use it to build ports instead of doing so manually.

Building / installing X.org with portmaster

Let’s say we want to install a minimal X11 environment on this machine. How can we do it? Actually as simple as telling portmaster the origin of the port to install:

# portmaster x11/xorg-minimal

Port options…

After running that command, you’ll find yourself in a familiar situation: Lots and lots of configuration menus for port options will be presented, one after another. Portmaster will recursively let you configure the port and all of its dependencies. In our case you’ll have 42 ports to configure before you can go on.

Portmaster in action

In the background portmaster is already pretty busy. It’s gathering information about dependencies and dependencies of dependencies. It starts fetching all the missing distfiles so that the compilation can start right away when it’s the particular port’s turn. All the nice little things that make building from ports a bit faster and a little less of a hassle.

Portmaster: Summary for building xorg-minimal (1/2)

Once it has all the required information gathered and sorted out, portmaster will report to you what it is about to do. In our case it wants to build and install 152 ports to fulfill the task we gave it.

Portmaster: Summary for building xorg-minimal (2/2)

Of course it’s a good idea to take a look at the list before you acknowledge portmaster’s plan.

Portmaster: distfile deletion?

Portmaster will ask whether to keep the distfiles after building. This is ok if you’re building something small, but when you give it a bigger task, you probably don’t want to sit and wait to decide on distfiles before the next port gets built… So let’s cancel the job right there by hitting Ctrl-C.

Portmaster: Ctrl-C report (1/2)

When you cancel portmaster, it will try to exit cleanly and will also give a report. From that report you can see what has already been done and what still needs to be done (in form of a command that would basically resume building the leftovers).

Portmaster: Ctrl-C report (2/2)

It’s also good to know that there’s the /tmp/portmasterfail.txt file that contains the command should you need it later (e.g. after you closed your terminal). Let’s start the building process again, but this time specify the -D argument. This tells portmaster to keep the distfiles. You could also specify -d to delete them if you need the space. Having told portmaster about your decision helps to not interrupt the building all the time.

# portmaster -D x11/xorg-minimal

Portmaster: Build error

All right, there we have an error! Things like this happen when working with older ports trees. This one is a somewhat special case. Texinfo depends on one file being fetched that changes over time but doesn’t have any version info or such. So portmaster would fetch the current file, but that has silently changed since the distfile information was put into our old ports tree. Fortunately it’s nothing too important and we can simply “fix” this by overwriting the file’s checksum with the one for the newer file:

# make makesum -C /usr/ports/print/texinfo
# portmaster -D x11/xorg-minimal

Manually fetching missing distfile

Next problem: This old version of mesa is no longer available in the places that it used to be. This can be fixed rather easily if you can find the distfile somewhere else on the net! Just put it into its place manually. Usually this is /usr/ports/distfiles, but there are ports that use subdirectories like /usr/ports/distfiles/bash or even deeper structures like e.g. /usr/ports/distfiles/xorg/xserver!

# fetch https://mesa.freedesktop.org/archive/older-versions/17.x/mesa-17.3.9.tar.xz -o /usr/ports/distfiles/mesa-17.3.9.tar.xz
# portmaster -D x11/xorg-minimal

Portmaster: Success!

Eventually the process should end successfully. Portmaster gives a final report – and our requested program has been installed!

What’s next?

Now you know how to build ports using portmaster. There’s more to that tool, though. But we’ll come to that in the next post which will also show how to use it to update ports.

FreeBSD: Building software from ports (2/2)

My previous post discussed what ports are, where they can be found on FreeBSD and what the files of which a port is composed of look like. This post will now detail how to use ports to build software on FreeBSD (the other BSDs have ports trees that work somewhat similar but are not identical. There are important differences!).

Packages and ports: A word of warning

The ports system works hand in hand with FreeBSD’s package manager Pkg. It makes little difference if some software on your machine was installed via a package or directly from ports – packages are in fact actually built from ports! Still it is not really recommended to mix packages and ports. In past times it was strongly discouraged. Things have changed since then. I’ve done it a lot – and mostly got away with it. Don’t rely on it, though, especially if you’re new to the whole topic. Feel free to do it on a test system and be completely happy – or face subtle and annoying breakage. You cannot know up front.

What’s the deal here? Modern software is a complex thing. Most programs rely on other programs or external libraries. A lot of programs can be configured at run time in certain ways. There are however decisions about program functionality that have to be made at compile time. The ports system allows you to build software with compile-time options other than the default. Pre-compiled packages have no chance to know that you choose to deactivate an option when you built a library yourself that they make use of. They assume that this feature is present (it was available on the system the package was built on after all!). And what can one poor program do in that case? Crash, explode, malfunction… A lot of things.

And then there’s the problem of mixing versions which can lead to all kinds of fun. If you stick with either ports or packages, you always have a consistent system with versions that are known to play together well (as long as the maintainers do their job well – we’re all humans and errors do occur).

Just keep that in mind when thinking about mixing programs installed from packages and ports on one system. You can do that. But it doesn’t mean you should. Enabling more options is generally safer than removing ones set by default. It can still have consequences. This is Unix though. Do whatever you see fit – and claim the responsibility. Your choice.

Most basic ports building

Building a software from ports is extremely easy. Go to the directory of a port and type make. Yes, that’s all! Let’s assume the port has no unsatisfied dependencies. The ports system will then check to see if the source code tarball is present in /usr/ports/distfiles. If it isn’t, it will automatically download it. Then it extracts the source code, prepares everything for the compilation and compiles it.

Building the ‘pkg’ port

On my fresh example system I build the Pkg manager from ports first – it’s needed for every other port anyway. Once everything has finished I get my shell back.

Building of Pkg completed

Installing the program is just as easy: Use make install

Installing the newly built port

That’s it, Pkg is now installed. We’re basically done with that port. However there’s still the “work” directory left over from the building process. To tidy up our port’s directory we can issue make clean.

Cleaning up after the build

Dependency handling

On to a just slightly more complex example. I want to build and install an old version of the LUA interpreter which depends on another port, libedit. Of course I could build devel/libedit first and then lang/lua51. In that case it wouldn’t be so bad. But if you think of larger programs with hundreds of dependencies that approach would be a nightmare.

So what to do about it? Well, nothing actually. The ports system takes care of it automatically! Just have it build LUA and it will figure out that it has to build the dependency first.

Building, installing and cleaning up in one command

The parameters to make that we used above are called make targets, BTW, and can be combined. That means it’s perfectly fine to issue make install clean together as you can see in the picture above.

Dependencies are handled automatically

The clean make target is also applied to all ports that were built as a dependency for the current port. Things like this make ports very convenient to use.

More on make and targets

Make targets can depend on other make targets. When you issue make install these are the targets that are actually run:

  • make config (more on that in a minute)
  • make fetch (fetch all files needed to build the port)
  • make checksum (check integrity of downloaded file(s))
  • make depends (check for missing dependencies and build/install those)
  • make extract (extract distfile(s) for the port)
  • make patch (apply patches for this port, if any)
  • make build (actually build the port)
  • make install (install the newly-built program)

If you type make checksum for example, all targets up to and including that one will run (that is config, fetch and checksum in that case). Running just make without any target will assume the default target which is equivalent to make build.

Also make will take an argument to look for the Makefile in another directory if you wish. So instead of doing e.g. this:

# cd /usr/ports/archivers/bzip2
# make install clean

you could also simply do this:

# make -C /usr/ports/archivers/bzip2 install clean

You’re in control: Ports options

So far it’s all nice and well but there’s no real advantage to using ports instead of packages. May I introduce ports options? Let’s say you we want to build BASH. If issue make in shells/bash, this is what happens:

Build options for BASH

The port ports-mgmt/dialog4ports is fetched and installed. It’s so small that you might miss it but it’s quite important. It’s needed to display the menu in the picture above which lets you set various options for the port.

You can now e.g. choose to not install the documentation if you’re short on space on a small or embedded system (sure, you wouldn’t actually compile on such a system, but that’s only an example, right?). If you don’t want BASH to support any foreign languages, deselect NLS. In case you feel that BASH’s built-in help is useless (did you ever issue the help command when you ran BASH?), you can cut that feature. Things like that.

If you see the option configuration for a port the first time, you see the default configuration. In general it’s a good idea to leave options alone if you’re in doubt what they do (do a little research if you have the time). Of course you’re also free to experiment with them. It’s your system.

Once you’re happy, accept your selection and the source tarball is being fetched, extracted, etc. You know the score.

Build options for bison

But what’s that? Another configuration menu (for bison)? And another (m4) and another (texinfo), etc… It’s 8 menus for a rather basic program like BASH! And worse: The building process will run and build dependencies and when a port with options is reached, the process is interrupted and prompts the user.

Now imagine you’re building a whole graphical desktop like MATE… Currently even the basic desktop would build no less that 338 dependency packages on a fresh system! And there’s quite a few ports on the list which build rather heavy software that takes it’s time compiling. It would totally make sense to let it build over night or at least not require you to keep staring at the screen, waiting for the next options selection to confirm, right?

Recursive operations

Exactly that’s why recursive operations are supported by the ports system. The standard make target that was implicitly run to open the options dialog is make config. The recursive option which would run the same on each and every port that’s listed as a dependency for the current port is make config-recursive.

If you want to build MATE as mentioned in the previous example, that would start a true marathon of options for you to configure. However it’s still a lot better to be able to do this up front so that the build process can run uninterruptedly afterwards.

Oh, and don’t be surprised if you went through it all only to find that still another configuration dialog pops up later! Why? Most likely you enabled an option on some package that made it depend on another package that’s not a dependency by default. And that package may need to have its options configured, too. So if you changed any options it makes sense to run make config-recursive again until no more new option dialog windows are displayed!

Recursively fetching distfiles for security/sudo

You can also do make fetch-recursive to fetch the distfiles for the current port and all dependencies. Again: Keep in mind that enabling more options may lead to new dependencies. If you want to make sure that you have all the distfiles, you might want to run make fetch-recursive again after changing ports options.

Other things to know

Wonder where the all the options are saved? They are stored in text files in /var/db/ports/category_portname. But there’s no need to edit or delete them; if you want to get rid of them, there’s make rmconfig to do that. Also make rmconfig-recursive exists if you feel like blowing away a huge amount of them.

Ports options in /var/db/ports

Another thing that comes in handy is make build-depends-list which will show you a list of ports that will be built as build dependencies for your current port. If you want to see the runtime dependencies you would use make run-depends-list. And then there’s also make all-depends-list which will show you each and every port that would be installed if you chose to build the current port.

Showing port dependencies

You should also know that you can deinstall a port by using make deinstall. Yes, it is also possible to remove the package using pkg delete but that will lead to a problem. The ports infrastructure keeps track of installed ports separately and Pkg does not know anything about this. So even if your package is removed, the Ports infrastructure will insist that it is still installed and there’s something very wrong with your system!

Now what to do if you have that case? Use make reinstall to install the package again even though ports thinks that it’s already installed.

More on ports?

To be honest, there’s quite a bit more to ports than I could cover here. You may want to man 7 ports to see what other targets are available and what they do. Also we haven’t even touched how to keep your system updated when using ports!

The ports infrastructure is a great means of installing customized programs on your system. It’s quite easy to use as you’ve seen. But things can be made even easier – which is why there are helper tools available. I will write a follow-up article covering those (not the next one, though). But for now enjoy all of those new possibilities with software on your FreeBSD machines!

FreeBSD: Building software from ports (1/2)

In my previous two (link) posts (link) I wrote about using Pkg, FreeBSD’s package manager.

Pre-built binary packages are convenient to use but sometimes you need some more flexibility, want an application that cannot not be distributed in binary form due to license issues (or have some other requirements). Building software by hand is certainly possible – but with all the things involved, this can be a rather tedious process. It’s also slow, error-prone and there’s often no clean way to get rid of that stuff again. FreeBSD Ports to the rescue!

This first part is meant as a soft introduction to FreeBSD’s ports, assuming no prior knowledge (if I fail to explain something, feel free to comment on this post). It will give you enough background information to understand ports enough to start using them in the next article.

What “Ports” are

When programmers talk about porting something over, what they originally meant is this: Take an application that was written with one processor architecture in mind (say i386) and modify the source so that it runs on another (arm64 for example) afterwards. The term “porting” is also used when modifying the source of any program to make it run on another OS. The version that runs on the other architecture/OS is called a port of the original program to a different platform.

FreeBSD uses the term slightly differently. There’s a lot of software written e.g. for Linux that will build and work on FreeBSD just fine as it is. Even though it does not require any changes, that software might be ported to FreeBSD. So in this case “porting” does not mean “make it work at all” but make it easily available. This is done by creating a port for any program. That term doesn’t mean a variant of the source code in this case but rather a means to give you easy access to that software on FreeBSD.

So what is a port in FreeBSD? Actually a port is a directory with a bunch of files in it. The heart of it is one file that basically is a recipe if you will. That recipe contains everything needed to build and install the port (and thus have the application installed on your machine in the end). Following this metaphor you could think of all the ports as a big cookbook. Formally it is known as the Ports collection. All those files in your filesystem related to ports are refered to as the Ports tree.

How to get the Ports tree

There are several options to obtain a copy of the ports tree. When you install FreeBSD you can decide whether or not to install it, too. I usually don’t do that because on systems that use binary packages only. It wastes only about 300 MB of space, but more importantly consists of almost 170.000 files (watch your inodes on embedded devices!). Take a look at /usr/ports: If that directory is empty your system is currently missing the ports tree.

The simplest way to get it is by using portsnap:

# portsnap fetch extract

If you want to update the tree later, you can use:

# portsnap fetch update

Another way is to use Subversion. This is more flexible: With portsnap you always get the current tree while Subversion also allows you to checkout older revisions, too. If you plan to become a ports developer, you will probably want to use Subversion for tools like svndiff. If you just want to use ports, portsnap should actually suffice. All currently supported versions of FreeBSD contain a light-weight version of Subversion called svnlite.

Here’s how to checkout the latest tree:

# svnlite checkout https://svn.freebsd.org/ports/head /usr/ports

If you want to update it later run:

# svnlite update /usr/ports

Old versions of the tree

You normally shouldn’t need these but it’s good to know that they exist. Using Subversion you can also retrieve old trees. Be sure that /usr/ports is empty (including for Subversion’s dot directories) or Subversion will see that there’s already something there and won’t do the checkout. If for example you want the ports tree as it existed in 2016Q4, you can retrieve it like this:

# svnlite checkout https://svn.freebsd.org/ports/branches/2016Q4 /usr/ports

There are also several tags available that allow to get certain trees. Maybe you want to see which ports were available when FreeBSD 9.2 was released. Get the tree like this:

# svnlite checkout https://svn.freebsd.org/ports/tags/RELEASE_9_2_0 /usr/ports

And if you need the last tree that is guaranteed to work with 9.x there’s another special tag for it:

# svnlite checkout https://svn.freebsd.org/ports/tags/RELEASE_9_EOL /usr/ports

Keep in mind though that using old trees is risky because they contain program versions with vulnerabilities that have since been found! Also mind that it’s NOT a smart thing to simply get the tree for RELEASE_7_EOL because it still holds a port for PHP 5.2 and you thought that it would be cool to offer your customers as many versions as possible. Yes, it may be possible that you can still build it if you invest some manual work. But no, that doesn’t make it a good idea at all.

Oh, and don’t assume that old ports trees will be of any use on modern versions of FreeBSD! The ports architecture changed quite a bit over time, the most notable change being the replacement of the old pkg_* tools with the new Pkg. Ports older than a certain time definitely won’t build in their old, unmodified state today (and I say it again: You really shouldn’t bother unless you have a very special case).

Port organization

Take a look at the contents of /usr/ports on a system that has the tree installed. You will find over 60 directories there. There are a few special ones like distfiles (where tarballs with program’s source code get stored – might be missing initially) or Mk that holds include files for the ports infrastructure. The others are categories.

If you’re looking for a port for Firefox, that will be in www. GIMP is in graphics and it’s probably no surprise that Audacious (a music player) can be found in audio. Some program’s categories will be less obvious. LibreOffice is in editors which is not so bad. But help2man for example is in misc and not in converters or devel as at least I would expect if I didn’t know. In general however after a while of working with ports you will have a pretty good chance to guess where things are.

Say we are interested in the port for the window manager Sawfish for example. It’s located in /usr/ports/x11-wm/sawfish. Let’s take a closer look at that location and take it apart:

/usr/ports is the “ports directory”.
x11-wm (short for X11 window managers) is the category.
sawfish is the individual port’s name.

When referring to where a port lives, you can omit the ports directory since everybody is assumed to know where it is. The important information when identifying a port is the category and the name. Together those form what is known as the port origin (x11-wm/sawfish in our case).

How to find a port in the tree

There are multiple methods to find out the origin for the port you are looking for. Probably the simplest one is using whereis. If we didn’t know that sawfish is in x11-wm/sawfish we could do this:

% whereis sawfish
sawfish: /usr/ports/x11-wm/sawfish

This does however only work if you know the exact name of the port. And there’s a little more to it: Sometimes the name of a port and a package differ! This is often the case for Python-based packages. I have SaltStack installed, for example. It’s a package called py27-salt:

% pkg info -x salt
py27-salt-2017.7.1_1

If we were to look for that, we wouldn’t find it:

% whereis py27-salt
py27-salt:

So where is the port for the package?

% pkg info py27-salt
py27-salt-2017.7.1_1
Name           : py27-salt
[...]
Origin         : sysutils/py-salt
[...]

Here you can see that the port’s name is py-salt! The “27” gets added when the package is created and reflects the version of Python that it’s build against. You may also see some py3-xyz ports. In those cases the name reflects that the port cannot be built with Python 2.x. The package will still be called py36-xyz, though (or whatever the default Python 3.x version is at that time)!

When discussing package management I recommended FreshPorts and when working with ports it can be useful, too. Search for some program’s name and it might be easier for you to find the package name and the port origin for it!

What a port looks like

Let’s take a look at the port for the zstd compression utility:

% ls /usr/ports/archivers/zstd/
distinfo	Makefile	pkg-descr	pkg-plist

So what have we here? The simplest file is pkg-descr. Each package has a short and a long package description – this file is what contains the latter: A detailed description that should give you a good idea whether this port would satisfy your needs:

% cat /usr/ports/archivers/zstd/pkg-descr
Zstd, short for Zstandard, is a real-time compression algorithm providing
high compression ratios.  It offers a very wide range of compression vs.
speed trade-offs while being backed by a very fast decoder.  It offers
[...]

Then there’s a file called distinfo. It lists all files that need to be downloaded to build the port (usually the program’s source code). It also contains a checksum and the file’s size to make sure that the valid file is being used (an archive could get corrupted during the transfer or you could even get an archive that somebody tempered with!):

% cat /usr/ports/archivers/zstd/distinfo 
TIMESTAMP = 1503324578
SHA256 (facebook-zstd-v1.3.1_GH0.tar.gz) = 312fb9dc75668addbc9c8f33c7fa198b0fc965c576386b8451397e06256eadc6
SIZE (facebook-zstd-v1.3.1_GH0.tar.gz) = 1513767

There’s usually also pkg-plist. It lists all the files that are installed by the port:

% cat /usr/ports/archivers/zstd/pkg-plist 
bin/unzstd
bin/zstd
bin/zstdcat
[...]
lib/libzstd.so.%%PORTVERSION%%
libdata/pkgconfig/libzstd.pc
man/man1/unzstd.1.gz
man/man1/zstd.1.gz
man/man1/zstdcat.1.gz

And finally there’s the Makefile. This is where all the magic happens. If you’re a programmer or you have built software from source before, there’s a high chance that you’re at least somewhat familiar with a tool called make. It processes Makefiles and then does as told by those. While it’s most often used to compile software it can actually be used for a wide variety of tasks.

If you don’t have at least some experience with them, Makefiles look pretty much obscure and creating them seems like a black art. If you’ve ever looked at a complicated Makefile, you may be worried to hear that to use ports you have to use make. Don’t be. The people who take care of the Ports infrastructure are the ones who really need to know how to deal with all the nuts and bolts of make. They’ve already solved all the common tasks so that the porters (those people who create the actual ports) can rely on it. This is done by including other Makefiles and it manages to hide away all the scariness. And for you as a user things are even simpler as you can just use what others created for you!

Let’s take a look at the Makefile for our example port:

% cat /usr/ports/archivers/zstd/Makefile 
# Created by: John Marino <marino@FreeBSD.org>
# $FreeBSD: head/archivers/zstd/Makefile 448492 2017-08-21 20:44:02Z sunpoet $

PORTNAME=	zstd
PORTVERSION=	1.3.1
DISTVERSIONPREFIX=	v
CATEGORIES=	archivers

MAINTAINER=	sunpoet@FreeBSD.org
COMMENT=	Zstandard - Fast real-time compression algorithm

LICENSE=	BSD3CLAUSE GPLv2
[...]
post-patch:
	@${REINPLACE_CMD} -e 's|INSTALL_|BSD_&|' ${WRKSRC}/lib/Makefile ${WRKSRC}/programs/Makefile

.include <bsd.port.mk>

Now that doesn’t look half bad for a Makefile, does it? In fact it’s mostly just defining Variables! The only line that looks somewhat complex is the “post-patch” command (which is also less terrifying than it first looks – if you know sed you can surely guess what it’ll do).

There can actually be more files in some ports. If FreeBSD-specific patches are required to build the port, those are included in the ports tree. You can find them in a sub-directory called files located in the port’s directory. Here’s an example:

% ls /usr/ports/editors/vim/files/
patch-src-auto-configure        vietnamese_viscii.vim
patch-src-installml.sh          vimrc

The patches there are named after the files that they apply to. Every patch in the files directory is automatically applied when building the port.

What’s next?

Alright. With that we’ve got a basic overview of what Ports are covered. The next post will show how to actually use them to build and install software.