Advance!BSD – thoughts on a not-for-profit project to support *BSD (1/2)

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/advance_bsd_pt1.gmi

There are multiple reasons why I am a BSD user and enthusiast. For one thing I enjoy using systems where the design benefits from the holistic approach (whole OS vs. “kernel + some packages”). I’ve come to appreciate things like good documentation, preference of simplicity over unnecessary complexity and permissive licensing. There’s a different spirit to the community that makes me feel more at home. Well, and truth be told: I do have a heart for “outsiders” who are doing well but get far less attention due to the towering popularity of a certain other system. In addition to that I’m very much concerned about the new de-facto monopoly in open source.

While the broader BSD community is far less fragmented than what you know from the Linux ecosystem, it’s also much smaller. Considering the little manpower that the BSD projects have, they are doing an outstanding job. But a lot of people seem to agree on the fact that due to the small amount of resources available, the BSDs are pretty far away from maximizing their full potential.

There are in fact a lot of people out there who’d like to help improve the situation. But coordination of volunteer effort is hard. Linux is what Linux is for a substantial part due to corporate funding. And while there are also companies that support BSD, the question is: Could we probably do better?

A (nonprofit) BSD-first service provider?

After thinking about this for quite a while, I finally just asked on Reddit what other people think about such a project – and I’ve been blown away by the response!

While I had hoped that most people would agree that this could be an interesting thing or even consider supporting it, I had not anticipated that the most popular option on the poll would be the one where people express their interest in the project with the prospect of perhaps participating actively in getting it started!

20 votes (after only one day) for maybe participating in the project!

A lot of projects struggle for years to find people who might be willing to join. With projects that weren’t even started yet, and thus have nothing to show off, it’s even harder to get some attention. Getting 20 people to support the cause in just one day was quite a surprise for me. Sure, that’s only a poll vote on Reddit and thus completely without any obligation. But let’s assume that 1/4 would actually join such a project and contribute – 5 people is not bad at all. Depending on what skills they bring in, even 2 or 3 might suffice to get something started, further increasing the odds that more people join in. The hardest part in getting a project on they way is finding people who are willing to pioneer it and make something that sounds interesting actually work well.

Why “Advance!BSD”?

The name is just a working title, BTW. I won’t insist on it and am in fact pretty sure that something better could easily be found.

Recently there have been two longer discussions (also on Reddit) about what the BSDs lack to be more competitive. There were a lot of great ideas and I’m one of the people who’d like to see at least some of them being implemented eventually. But let’s be realistic: This is not very likely to happen. There’s enough work going on regarding the must-haves, and when developers decide to work on a nice-to-have, it’ll be something that they want. If we really want to see some things not of very high priority to the various projects eventually land, we need to take care of that ourselves. If you’re a developer with the required skills and enough free time: Great! Nobody can really stop you. If you aren’t: Too bad.

I’ve thought about the possibilities of crowd-funding that we have today. In theory we could try to find a enough people who’d like to see a feature and are willing to spend some money so that the group could contract a developer. Even though I believe that there are enough developers out there who’d love to do some paid work on *BSD and would probably even prefer such a task over one that pays better, I’m not very optimistic about it. There’s not such a high chance of finding people who want the same feature implemented and would be willing to pay for it. Certainly not over a longer time which would be required to collect enough money. Also I don’t really like that begging for people’s money approach, either. So I came up with a different idea:

If you like *BSD you’re almost certainly an IT person. IT people all have a couple of things in common: We love tech. We use email. Almost everybody has his or her own homepage. Which means: We need domains. We need DNS. We need Webspace, etc. And we get that from service providers. Some of which are somewhat BSD-friendly, many of which are not. And even cloud providers for example that offer BSD as an option usually don’t do that because they love it (and regularly that shows…).

So what about providing services instead of just asking for money? Imagine starting a “hosting club” where like-minded people get something rolling that works for them and that could at some point be turned into a not-for-profit hosting provider that lives and breathes BSD! The latter would use the money acquired to pay the running costs and spend the rest on improving the BSDs. What could that look like?

Well, for example like this: Members could vote for a general topic for each year (like e.g. “desktop”, “drivers”, “de-GPLing”, “porting efforts”, …) and propose (as well as discuss) concrete project ideas over the year then voting again at the end of the year when the available money is to be spent. Any project that is beneficial for more than one specific BSD gets bonus points.

Which BSD operating system(s)?

In a follow-up poll I asked people interested in the project about which BSDs they are proficient with. Considering the general market share within *BSD it came to no surprise:

The largest group of votes were for FreeBSD somewhat closely followed by OpenBSD. People voting NetBSD, DragonFly BSD or multiple followed only after a huge gap.

Most people favor FreeBSD and OpenBSD

Pretty early on one person commented that even though NetBSD was his favorite OS, it might make most sense to go with FreeBSD for such a project for example due to jails being a very useful feature for that. I can imagine that there might be more people who decided to put their personal preferences aside and vote for what they think makes most sense for this use case.

To be honest, I’d like to see all of the BSDs being part of this in some way or another. But to be realistic, I think we need to start with something. This should be one or two BSD systems that project members are familiar with. Right now FreeBSD and OpenBSD seem to be the reasonable choices here. Of course we need to take into account what those systems excel in and which one to use for what.

Problems with the not-for-profit status

Like with everything, there’s pros and cons regarding the aim for this to be a not-for-profit eventually.

Pro:

It ensures that nobody involved in it could ever become greedy and try to sabotage the original idea of providing money to support *BSD. It would also protect such an organization from becoming attractive for buyout by a for-profit competitor should it go well. There would be benefits regarding taxes. And I’d imagine that it gives a good feeling to the customers which could be turned into a competitive advantage.

Contra:

The price to pay is inflexibility. A not-for-profit can donate money only to another not-for-profit organizations – and that very likely only in the country that it was formed in. With e.g. the FreeBSD foundation and the OpenBSD foundation we have to potential organizations that we might want to donate to; however one is US-based while the other one is Canadian. A for-profit company is free to spend money however it wishes. There might be other limitations that I’m not aware off. Going with not-for-profit would require consulting lawyers upfront.

A federated model?

One interesting idea that came up on Reddit was that of possibly going with a federated model. In this case project members / supporters who own a server but don’t need all of its resources would dedicate some percentage of CPU, memory and disk space for a VM that could be used for the project. The user who suggested this envisions that there could be some sort of market place where people can offer those resources to people who want a VM. The person who donates the resources gets to decide what field the biggest part of the money made from this would be spent on. The customer on the other hand can also influence the the direction by picking e.g. VM a) where the money goes to desktop improvements over option b) were he’d support permissively licensed alternatives.

I like this idea and think that we should discuss it. It has the obvious benefit of being able to start without somewhat high upfront costs for hardware. Also more people might be ok with a “donate some spare resources” approach for machines that are running anyway. The downside certainly is: Reliability. Donated resources might basically be withdrawn anytime. What could / should we do about that?

What’s next?

While this part 1 mostly covered the “why”, part two of the article is more about the “what” and the “how”.

FreeBSD package building pt. 5: Sophisticated Synth

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/freebsd_package_building_pt5.gmi

In the previous posts of this series, there was an introduction to package building on FreeBSD and we discussed basic Synth usage. The program’s configuration, working with compiler cache and using the tool to update the installed applications was covered as well. We also discussed Synth web reports, serving repositories over HTTP and building package sets. We also took a brief look at the logs to find out why some ports failed.

In this article we are going to sign our package repository and explore make.conf options. We will also add an additional profile to build packages for an obsolete FreeBSD release that is 32-bit and automate package building with cron.

Changing the structure

So far we’ve only considered building basically one local package set (even though we’ve shared it). If we want to have a single build server manage multiple package sets, we will need a somewhat more complex directory structure to allow for separate repositories and such. Should you be building on a UFS system, it’s easy: Just create the additional directories. I’m using ZFS here, though, and need to think if I want to create the whole structure as several datasets or just two single datasets with custom mount? As usually there’s pros and cons for both. I’m going with the former here:

# rm -r /var/synth
# zfs create zroot/var/synth
# zfs create zroot/var/synth/www
# zfs create zroot/var/synth/www/log
# zfs create zroot/var/synth/www/log/13.0_amd64
# zfs create zroot/var/synth/www/packages
# zfs create zroot/var/synth/www/packages/13.0_amd64
# synth configure

Now we need to adapt the Synth configuration for the new paths:

B needs to be set to /var/synth/www/packages/13.0_amd64 and E to /var/synth/www/log/13.0_amd64. Normally I’d create a custom profile for that, but as I’m covering that a little later in this article, we’re going to abuse the LiveSystem default profile for now.

Next is re-configuring the webserver:

# vi /usr/local/etc/obhttpd.conf

Remove the block return directive in the location “/” block on the synth.local vhost and replace it with:

directory auto index

Then change the location to “/*”. I’m also removing the second location block. Create a new .htpasswd and bring over authentication to the main block if you want to.

# service obhttpd restart

Repository signing

To be able to use signing, we need a key pair available to Synth. Use the openssl command to create a private key, change permissions and then create a public key, too:

# openssl genrsa -out /usr/local/etc/synth/LiveSystem-private.key 2048
# chmod 0400 /usr/local/etc/synth/LiveSystem-private.key
# openssl rsa -pubout -in /usr/local/etc/synth/LiveSystem-private.key -out /usr/local/etc/synth/LiveSystem-public.key

Mind the filenames here! The LiveSystem part refers to the name of the profile we’re using. If you want to sign different repositories resulting from various profiles, make sure that you place the two key files for each of the profiles in /usr/local/etc/synth.

While you’re at it, consider either generating a self-signed TLS certificate or using Let’s Encrypt (if you have own a proper domain). If you opted to use TLS, change the webserver configuration once more to have it serve both the log and the package vhosts via HTTPS. There’s an example configuration (obhttpd.conf.sample) that comes with obhttpd in case you want to take a look. It covers HTTPS vhosts.

Alright! Since we changed the paths, we don’t currently have any repository to sign. Let’s build a popular browser now:

# synth build www/firefox

Firefox failed in the configure phase!

Firefox failed to build. This is what the log says:

DEBUG: Executing: `/usr/local/bin/cbindgen --version`
DEBUG: /usr/local/bin/cbindgen has version 0.18.0
ERROR: cbindgen version 0.18.0 is too old. At least version 0.19.0 is required.

Please update using 'cargo install cbindgen --force' or running
'./mach bootstrap', after removing the existing executable located at
/usr/local/bin/cbindgen.

===>  Script "configure" failed unexpectedly.
Please report the problem to gecko@FreeBSD.org [maintainer] and attach the
"/construction/xports/www/firefox/work/.build/config.log" including the output
of the failure of your make command. Also, it might be a good idea to provide
an overview of all packages installed on your system (e.g. a
/usr/local/sbin/pkg-static info -g -Ea).
*** Error code 1

Stop.
make: stopped in /xports/www/firefox



--------------------------------------------------
--  Termination
--------------------------------------------------
Finished: Friday, 11 JUN 2021 at 03:03:06 UTC
Duration: 00:03:06

Oh well! We’ve hit a problem in the ports tree. Somebody updated the Firefox port in our branch to a version that requires a newer cbindgen port than is available in the same branch! Breakage like this does happen sometimes (we’re all human after all). What to do about it? In our case: Ignore it as it’s only an example. Otherwise I’d advise you to update to a newer ports tree as these problems are usually quickly redeemed.

Synth is asking whether it should rebuild the repository. Yes, we want to do that. Then it asks if it should update the system with the newly built packages. And no, not now. Also note: The synth build command that we used here, is interactive and thus not well fit if you want to automate things:

Would you like to rebuild the local repository (Y/N)? y
Stand by, recursively scanning 1 port serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Would you like to upgrade your system with the new packages now (Y/N)? n

What else do we have to do to sign the repository? Nothing. Synth has already done that and even changed the local repository configuration to make the package manager verify the signature:

# tail -n 3 /usr/local/etc/pkg/00_synth.conf
  signature_type: PUBKEY,
  pubkey        : /usr/local/etc/synth/LiveSystem-public.key
}

That wasn’t so hard now, was it? You might want to know that Synth also supports using a signing server instead of signing locally. If this is something you’re interested in, do a man 1 synth and read the appropriate section.

Global options with make.conf

FreeBSD has two main configuration files that affect the compilation process when using the system compiler. The more general one, /etc/make.conf and another one that is only used when building FreeBSD from source: /etc/src.conf. Since we’re talking about building ports, we can ignore the latter.

There’s a manual page, make.conf(5), which describes some of the options that can be put into there. Most of the ones covered there are only relevant for building the system. Do by all means leave things like CFLAGS alone if you don’t know what you’re doing! Regarding the ports tree, it’s most useful to set or unset common options globally. It’s very tedious to set all the options for your ports manually like this:

# make -C /usr/ports/sysutils/tmux config-recursive

You need to do this for specific ports that you want to change the options for. But if there’s some that you have a global policy for, it’s better to use make.conf. Let’s say we want to never include documentation and examples in our ports. This would be done by adding the following line to /etc/make.conf:

OPTIONS_UNSET+=DOCS EXAMPLES

This affects all ports whether built by Synth or not as well as each and every Synth profile. Let’s say we also want no foreign language support in our packages for the default Synth profile (but in all others), we’d create the file /usr/local/etc/synth/LiveSystem-make.conf and put the following in there:

OPTIONS_UNSET+=NLS

That setting will build packages without NLS in addition to building without DOCS and EXAMPLES – if “LiveSystem” is the active profile.

If you want to build all ports that support it with e.g. the DEBUG option, add another line:

OPTION_SET+=DEBUG

Some common options that you might want to use include:

  • X11
  • CUPS
  • GTK3
  • QT5
  • JAVA
  • MANPAGES

After unsetting DOCS and EXAMPLES globally as well as NLS for the default profile, we’re going to rebuild the installed packages next:

# synth prepare-system

Package rebuild done

Note that Synth only rebuilt the packages that were affected by the changed options either directly or via their dependencies. For that reason only 219 of the 344 packages actually installed were rebuilt. If we now use pkg upgrade, this is what happens (see screenshot):

pkg upgrade will free space due to removing docs, examples and nls files

Ignore the 3 packages getting updated; this is due to packages that were skipped due to the Rust failure originally. That port has successfully built when we were trying to build Firefox, so our latest package runs built three more packages that have not been updated to, yet.

More interestingly: There’s 31 reinstalls. Most of them due to the package manager detecting changed options and one due to a change to a required shared library. It’s not hard to do the math and figure out that 31 is quite a bit less than 219. It’s a little less obvious that build-time dependencies count towards that greater number while they don’t appear in the count of packages that are eventually reinstalled. Still it’s true that Synth takes a “better safe than sorry” approach and tends to rebuild some packages that pkg(8) will end up not reinstalling. But this is not much of a problem, especially if you’re using ccache.

Alternative profiles

If you want to use Synth to build more than one specific set of packages for exactly one platform, you can. One way to achive this would be to always change Synth’s configuration. But that’s tedious and error prone. For that reason profiles exist. They allow you to have multiple different configurations available at the same time. If you’re simply running Synth from the command line like we’ve always done so far, it will use the configuration of the active profile.

To show off how powerful this is, we’re going to do something a little special: Building 32-bit packages for the no longer supported version of FreeBSD 12.1. Since amd64 CPUs are capable of running i386 programs, this does not even involve emulation. We need to create a couple of new directories first:

# mkdir /var/synth/www/packages/12.1_i386
# mkdir /var/synth/www/log/12.1_i386
# mkdir -p /var/synth/sysroot/12.1_i386

The last one is something that you might or might not be familiar with. A sysroot is a somewhat common term for – well, the root of a system. The sysroot of our running system is /. But we can put the data for other systems somewhere in our filesystem. If we put the base system of 32-bit 12.1-RELEASE into the directory created last, that’ll be a sysroot for 12.1 i386. Technically we don’t need all of the base system and could cherry-pick. It’s easier to simply use the whole thing, though:

# fetch -o /tmp/12.1-i386-base.txz http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/i386/12.1-RELEASE/base.txz
# tar -C /var/synth/sysroot/12.1_i386 -xf /tmp/12.1-i386-base.txz
# rm /tmp/12.1-i386-base.txz
# synth configure

Alright. Now we’re going to create an additional profile. To do so, press > (greater than key), then chose 2 (i.e. Create new profile) and give it a name like e.g. 12.1-i386. Then change the following three settings:

B: /var/synth/www/packages/12.1_i386
E: /var/synth/www/log/12.1_i386
G: /var/synth/sysroot/12.1_i386

That’s all, after you save the configuration you’re ready to go. Create a list of packages you want to build and let Synth do it’s thing:

# synth just-build /var/synth/pkglist.12.1_i386

The build will fail almost immediately. Why? Let’s take a look. Building pkg(8) failed and here’s why:

--------------------------------------------------------------------------------
--  Phase: check-sanity
--------------------------------------------------------------------------------
/!\ ERROR: /!\

Ports Collection support for your FreeBSD version has ended, and no ports are
guaranteed to build on this system. Please upgrade to a supported release.

No support will be provided if you silence this message by defining
ALLOW_UNSUPPORTED_SYSTEM.

*** Error code 1

Stop.
make: stopped in /xports/ports-mgmt/pkg

Ok, since we’re trying to build for a system that’s not supported anymore, the ports infrastructure warns us about that. We have to tell it to ignore that. How do we do that? You might have guessed: By using a make.conf for the profile we’re using for this set of packages:

# echo ALLOW_UNSUPPORTED_SYSTEM=1 > /usr/local/etc/synth/12.1-i386-make.conf

Then try again to build the set – and it will just work.

Successfully built all the packages using the i386 profile

Automation & Hooks

Last but not least let’s put everything we’ve done so far together to automate building two package sets. We can make use of cron(8) to schedule the tasks. Let’s add the first one to /etc/crontab like this:

0	23	*	*	7	root	env TERM=dumb /usr/local/bin/synth just-build /var/synth/pkglist-12.1-i386

What does it do? It will run synth at 11pm every sunday to build all of the packages defined in the package list referenced there. There’s two things to note here:

  1. You need to disable curses mode in all the profiles you’re going to use. Synth still expects to find the TERM environment variable to figure out the terminal’s capabilities. You can set it to dumb as done here or to xterm or other valid values. If you don’t set it at all, Synth will not run.
  2. The cron entry as we’re using it here will use the active profile for Synth. It’s better to explicitly state which profile should be used. Let’s add another line to crontab for building the amd64 packages for 13.0 on Friday night:
0	23	*	*	5	root	env TERM=dumb SYNTHPROFILE=LiveSystem /usr/local/bin/synth just-build /var/synth/pkglist-13.0-amd64

In general I’d recommend to consider not calling synth directly from cron but to write small scripts instead. You could for example backup the current package set before actually starting the new build or you could snapshot the dataset after the successful build and zfs-send it off to another system.

One last thing that you should be aware of is that Synth provides hooks like hook_run_start, hook_run_end, hook_pkg_failure and so on. If you’re considering using hooks, have a look at the Synth manpage, they are covered well there.

What’s next?

Next topic would be covering Poudriere. However I’m considering taking a little break from package building and writing about something else instead before returning to this topic.

FreeBSD package building pt. 4: (Slightly) Advanced Synth

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/freebsd_package_building_pt4.gmi

In the previous posts of this series, there was an introduction to package building on FreeBSD and we discussed basic Synth usage. The program’s configuration, working with compiler cache and using the tool to update the installed applications was covered as well.

This article is about some advanced functionality of Synth like using web reports, building package sets and serving repositories as well as taking a look at logs.

Reporting

Synth comes with very useful reporting reporting functionality. For the default LiveSystem profile, Synth writes its logs to /var/log/synth. There it also creates a subdirectory called Reports and puts an advanced web report in there. It looks like this:

% ls -1 /var/log/synth/Report 
01_history.json
favicon.png
index.html
progress.css
progress.js
summary.json
synth.png

We’re going to build and setup a web server to make it accessible. I will use OpenBSD HTTPd for a reason that we’re going to talk about in a minute (besides me liking it quite a bit). Let’s use Synth to install it first. Then we’re going to enable it and create a log directory for it to use:

# synth install www/obhttpd
# sysrc obhttpd_enable=YES
# mkdir /var/log/obhttpd

OpenBSD HTTPd successfully installed by Synth

Alright. Remember how I told you not to change Synth’s directory configuration unless you have a reason for it? Now we have: We’re going to serve both the web report and the packages over http and we’re using OpenBSD HTTPd for that. That browser is chrooted: Not just by default, BTW but by design! You cannot turn it off. Unless told otherwise, Synth has a directory structure that doesn’t fit this scenario well. So we’re going to change it.

First thing is creating a new directory for the logs and then changing the configuration so Synth uses that:

# mkdir -p /var/synth/www/log
> # synth configure

choc

Change setting E to /var/synth/www/log and save. The new directory will of course be empty. We could copy the old files over, but let’s just rebuild HTTPd instead. Using synth build or synth just-build doesn’t work, though; the tool will detect that an up to date package has already been built and that nothing needs to be done. That’s what the force command is handy for:

# synth force www/obhttpd

Web server setup for reports

Now that we have something to serve, we can edit the webserver’s configuration file. Simple delete everything and put something like this into /usr/local/etc/obhttpd.conf:

chroot "/var/synth/www"
logdir "/var/log/obhttpd"

server synth.local {
    listen on * port 80
    root "/log"
    log style combined
    location "/" {
        block return 302 "http://$HTTP_HOST/Report/index.html"
    }
}

This defines where the chroot begins and thus where the unprivileged webserver processes are contained. It also defines the log directory as well as a virtual host (simply called “server” in OpenBSD HTTPd) with the name “synth.local”. Either replace this with a proper domain name that fits your scheme and configure your DNS accordingly or use /etc/hosts on all machines that need to access it to define the name there.

The server part is pretty simple, too. It makes HTTPd bind on port 80 on every usable network interface. The web root is defined relative to the chroot, so in this case it points to /var/synth/www/log. I’ve grown a habit of using the more detailed combined log style; if you’re fine with the default common format, you can leave the respective line out. Finally the configuration block defines a special rule for location / which means somebody accesses the virtual host directly (i.e. http://synth.local in this case). It will make the browser be redirected to the report index instead. Getting a special file (like e.g. lang___python37.log in the log directory) will not trigger the rule and thus still work. This is just a convenience thing and if you don’t like it leave it out.

All that’s missing now is starting up the webserver:

# service obhttpd start

You should now be able to point your browser at the the vhost’s name (if you made it resolve). Just using the machine’s IP address is also going to work in this case since it’s the default vhost. But better make it reachable using the configured name as we’re adding another vhost in a moment.

Synth web report for the latest package run

Authentication

But for now what about security? Let’s say you don’t want to share your report with the whole world. One easy means of protecting it is by using HTTP basic auth. OpenBSD HTTPd uses standard .htpasswd files. These can however use various cryptographic hashes for the passwords – whereas HTTPd only supports one: bcrypt.

The first time I tried to do authentication with OpenBSD HTTPd, it drove me completely nuts as I couldn’t get it working. Fortunately I own Michael W. Lucas’ excellent book “Httpd and Relayd Mastery”. After digging it out the index indicated that I might want to read page 28. I did, banged my head against the table and immediately got it working using that hashing algorithm. Don’t be like me, skip trying to use foreign tools you may be used to and just do it right in the first place. HTTPd comes with its own htpasswd binary. Use that.

In this example I’m adding a user called “synth”. Use whatever you prefer. Then give the password two times. This leaves you with a valid .htpasswd file that HTTPd could use – if it was allowed to access it! Let’s fix that problem:

# chown root:wheel /var/synth/www/.htpasswd
> # chmod 0640 /var/synth/www/.htpasswd

Having the authentication information in place, we only need to add another location block to the webserver’s vhost configuration. Put the following in there after the line that closes the previous location block:

    location "/Report/* {
        authenticate with "/.htpasswd"
    }

Note the htpasswd file’s location! It’s within the chroot (or it couldn’t be accessed by the webserver), but outside the webroot directory. So HTTPd could never accidentally serve it to somebody who knew that it was there and requested the file.

The only thing that remains is restarting the webserver. Next time you visit the report page, you’ll be asked to authenticate first.

# service obhttpd restart

Package repository

So far all of our packages have been created in a directory outside of the webserver’s chroot. If we want to make them available via HTTP, we need to use another path for them. Therefore we’re going to create a directory and reconfigure Synth again:

# mkdir -p /var/synth/www/packages
# synth configure

This time it’s setting B. Change it to /var/synth/www/packages and save. Now let’s build a package that draws in a couple of dependencies:

# synth just-build chocolate-doom

We can watch it now via the web reports while it’s building. Since it’s a new directory where no packages exist, yet, Synth is first going to build the package manager again. During this early stage no report is available, but once that’s finished the reports work.

While we’re rebuilding all packages due to the new package directory, Synth can take advantage of ccache as we haven’t modified it’s path. Wonder how much of a difference that actually makes? Building llvm10 on its own, one time using the cache and one time (for testing purposes) without it will show you the difference:

Duration: 00:13:32 (with ccache)
Duration: 02:09:37 (uncached)

Synth web report while it’s building

It gives us all the information that the curses UI holds – and more. The number of entries for completed packages can be changed. You can browse those page-wise back to the very first packages. It’s possible to use filters to e.g. just list skipped or failed packages. You can view / download (whatever your browser does) the log files for all those packages. And there’s even a search (which can be very handy if you’re building a large package set).

Report with only 10 entries per page

As long as packages are being built, the report page also shows the builder information and automatically refreshes the page every couple of seconds. Once it completes, it removes builder info (which would only waste space) and stops the polling. You can always come back later and inspect everything about the latest package run. The next one will overwrite the previous information, though.

Synth report search function

Now that we have a bunch of newly built packages, let’s see what that looks like:

# ls -1 /var/synth/packages/All
autoconf-2.69_3.txz
autoconf-wrapper-20131203.txz
automake-1.16.3.txz
binutils-2.33.1_4,1.txz
bison-3.7.5,1.txz
ca_root_nss-3.63.txz
ccache-3.7.1_1.txz
celt-0.11.3_3.txz
chocolate-doom-3.0.1.txz
cmake-3.19.6.txz
curl-7.76.0.txz
db5-5.3.28_7.txz
docbook-1.5.txz
docbook-sgml-4.5_1.txz
docbook-xml-5.0_3.txz
docbook-xsl-1.79.1_1,1.txz
doom-data-1.0_1.txz
evdev-proto-5.8.txz
expat-2.2.10.txz
flac-1.3.3_1.txz
[...]

Showing only ignored packages in the report (none in this case)

The packages are there. But what’s in the main directory?

# ls -l /var/synth/www/packages
total 18
drwxr-xr-x  2 root  wheel  150 Jun  7 23:57 All
drwxr-xr-x  2 root  wheel    3 Jun  7 23:21 Latest

This is not a valid pkg(8) repository. Which is no wonder since we used just-build. So we’re going to have Synth create an actual repository from these packages next:

Searching in the report after the build was completed

# synth rebuild-repository
# ls -l /var/synth/www/packages
total 117
drwxr-xr-x  2 root  wheel    150 Jun  7 23:57 All
drwxr-xr-x  2 root  wheel      3 Jun  7 23:21 Latest
-rw-r--r--  1 root  wheel    163 Jun  8 00:02 meta.conf
-rw-r--r--  1 root  wheel    236 Jun  8 00:02 meta.txz
-rw-r--r--  1 root  wheel  40824 Jun  8 00:02 packagesite.txz

Here we go, that’s all that pkg(8) needs. Synth should have automatically updated your repository configuration to use the new location. Have a look at /usr/local/etc/pkg/repos/00_synth.conf – the URL should point to the new directory.

Serving the repository

The next step is to make the repository available in the network, too. So edit /usr/local/etc/obhttpd.conf once more and add another “server” (i.e. vhost):

server pkg.local {
    listen on * port 80
    root "/packages"
    log style combined
    location * {
        directory auto index
    }
}

One service restart later you should be able to access the repository via a web browser from any machine in the same subnet (if you got your DNS right):

# service obhttpd restart

Looking at the package repository with a browser

This is already it, but let’s prove that it works, too. I’m adding the “pkg.local” name to the machine’s 127.0.0.1 definition in /etc/hosts, then change the URL in the Synth repo to fetch packages via HTTP:

  url      : http://pkg.local,

I’ve also created a FreeBSD.conf to disable the official repository. Let’s stop the webserver for a second and then try to update the package DB:

# service obhttpd stop
# pkg update
Updating Synth repository catalogue...
pkg: Repository Synth has a wrong packagesite, need to re-create database
pkg: http://pkg.local/meta.txz: Connection refused
Unable to update repository Synth
Error updating repositories!

Ok, so there’s no other repository configured anymore and this one is not accessed via the filesystem. So we’re going to start the webserver once more (give it a sec) and then try again:

# service obhttpd start
# pkg update
Updating Synth repository catalogue...
Fetching meta.conf: 100%    163 B  0.2kB/s    00:01
Fetching packagesite.txz: 100%   40 KiB  40.8kB/s   00:01
Processing entries: 100%
Synth repository update completed. 148 packages processed.
All repositories are up to date.

Great! So now we can install DooM on this box or on any other machine running FreeBSD 13.0 which can reach it over the network.

Package sets

So far we’ve only either built all packages for what was already installed on the machine or for single ports that we selected at the command line. But now that we can serve packages over the network, it’s rather tempting to use a powerful build machine to build packages for various other FreeBSD machines, isn’t it? Let’s assume that you’re going to share packages with a KDE lover.

First we should prepare a list of packages that we need, starting with what is installed on our machine.

# pkg info | wc -l
345

Wow, that’s already quite some packages for such a pretty naked system! But we don’t need to consider them all as most of them are just dependencies. Let’s ask pkg(8) for the origin of all packages we explicitly installed (i.e. which were not recorded as automatically installed):

# pkg query -e %a=0 %o > /var/synth/pkglist
# cat /var/synth/pkglist
x11-wm/awesome
devel/ccache
graphics/drm-kmod
devel/git
www/obhttpd
ports-mgmt/pkg
ports-mgmt/portmaster
x11/sakura
x11/setxkbmap
security/sudo
ports-mgmt/synth
sysutils/tmux
x11/xfce4-screenshooter-plugin
x11/xorg-minimal

That’s better! But we certainly don’t need portmaster anymore, so we can take it off the list (and deinstall it actually). Let’s add www/firefox and x11/kde5 for our pal (and sort the list since it’s a bit ugly right now).

Once that’s done, we should be able to do a simple:

# synth build /var/synth/pkglist
Error: port origin 'devel/git' not recognized.
Perhaps you intended one of the following flavors?
   - devel/git@default
   - devel/git@gui
   - devel/git@lite
   - devel/git@svn
   - devel/git@tiny

Oh yes, right! We need to edit our list and specify the flavor to build! I’m going with the lite variant here, so the git line needs to be changed to this:

devel/git@lite

Then we can try again – and yes, it starts building after calculating the required dependencies.

Logs

Whoopsie! After about an hour stupid me removed the network cable for a moment. This has caused a couple of build failures (see screenshot). The report will display the phase that the build failed. In this case it’s the fetch phase (and we don’t have to look for the reason as we already know it). Sometimes a distfile mirror is temporary down or the distfile has been removed. In that case you will have to manually get the files and put them into the distfiles directory. Skipped ports also display the reason, i.e. which dependency failed previously.

Failed and skipped ports due to a connection problem

I better re-attach that cable right away and start the building over… Many hours later it has finished. But what’s this? Rust has failed again (and this time it wasn’t me)! And it failed at the stage phase. When this happens it’s usually because of a broken port got committed. Update your ports tree and hope that it has been fixed in the meantime. This is not the reason in our case, however.

Another phase, another failure!

But how do we find out what actually happened? Well, by looking at the logs, of course. Here’s the last 15 lines of the relevant log:

        finished in 183.141 seconds
  < Docs { host: TargetSelection { triple: "x86_64-unknown-freebsd", file: None } }
Install docs stage2 (Some(TargetSelection { triple: "x86_64-unknown-freebsd", file: None }))
running: "sh" "/construction/xports/lang/rust/work/rustc-1.51.0-src/build/tmp/tarball/rust-docs/x86_64-unknown-freebsd/rust-docs-1.51.0-x86_64-unknown-freebsd/install.sh" "--prefix=/construction/xports/lang/rust/work/stage/usr/local" "--sysconfdir=/construction/xports/lang/rust/work/stage/usr/local/etc" "--datadir=/construction/xports/lang/rust/work/stage/usr/local/share" "--docdir=/construction/xports/lang/rust/work/stage/usr/local/share/doc/rust" "--bindir=/construction/xports/lang/rust/work/stage/usr/local/bin" "--libdir=/construction/xports/lang/rust/work/stage/usr/local/lib" "--mandir=/construction/xports/lang/rust/work/stage/usr/local/share/man" "--disable-ldconfig"
install: creating uninstall script at /construction/xports/lang/rust/work/stage/usr/local/lib/rustlib/uninstall.sh
install: installing component 'rust-docs'
###  Watchdog killed runaway process!  (no activity for 78 minutes)  ###



--------------------------------------------------
--  Termination
--------------------------------------------------
Finished: Wednesday, 9 JUN 2021 at 00:04:47 UTC
Duration: 04:27:03

Ha! The build process was killed by the watchdog! Bad doggy? It does happen that the process would eventually have finished. Not this time. We have to dig a little deeper. In /var/log/messages of the build machine I can find the messages kernel: swap_pager: out of swap space and kernel: swp_pager_getswapspace(4): failed. This machine has 24 GB of RAM and 8 GB of swap space configured. And by building 6 huge ports concurrently, it exceeded these resources! Keep in mind that package building can be quite demanding, especially if you use tmpfs (which you should if you can).

So, there we are. We’ve configured our build server for web reports and serving the repository. We’ve looked at building package sets and covered a few examples of what can go wrong. And that’s it for today.

What’s next?

The last article about Synth will cover make.conf, signing repositories and using cron for automated builds. We’ll also take a brief look at profiles.

FreeBSD package building pt. 3: Intermediate Synth

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/freebsd_package_building_pt3.gmi

In this article we’ll continue exploring Synth. After covering a general introduction to package building as well as Synth’s basic operation in the previous articles, we’re going to look at the program’s configuration and using it for updating the system.

The first thing to do here is to get rid of the old ports tree and replace it with a newer one so that some updates become available:

# zfs destroy zroot/usr/ports
# zfs create zroot/usr/ports
# git clone --depth 1 -b 2012Q2 https://git.freebsd.org/ports.git /usr/ports
# mkdir /usr/ports/distfiles
# synth status

Synth regenerates the flavor index again, then gives the output. This time it will not just show all packages as “new” but most are either updates or rebuilds.

Synth status with new ports tree

Configuration

Let’s look at how you can change Synth’s behavior next:

# synth configure

Synth’s configuration is pretty straight-forward. Options A to G configure the various directories that the program uses. Should you change anything here? If you know what you are doing and have special needs: Maybe. If you actually ask the question whether to change any such directory, the answer is no. Option H (Compiler cache) is a special one. We’ll get to that in a minute.

With I you set the number of builders that Synth uses. J means how many threads each builder may use; think make -j $SOMENUMBER. When compiling a single piece of software, it’s usually recommended to set the number of jobs equal to the machines core count + 1. Take a look at the screenshot above: Doing the math, we’re configuring for 24 cores here – on a system that has 8 (with Hyper Threading).

Synth’s configuration menu

Why does Synth choose to over-provision the resources so much? The reason is simple: It’s only over-provisioned when more than three builders are active at the same time. Often enough not all builders will be at the build stage (where the compiling happens) at the same time. Most other stages are much lighter on CPU – which would mean that you’re wasting available resources (and thus prolong the total build time). Also in the previous post you’ve seen that LLVM and Rust took hours to build (all four remaining builders were idle most of the time!). If the cap would have been lower, build times would have increased even more.

So what’s the best setting for builder count and max jobs? There’s no single answer to that. It depends on both your machine and on the set of ports that you build. Play with both values a bit and get a feeling what works best for you. Or leave it at the default that Synth calculated for your system which will probably work well enough.

Options K and L are real speed boosters. They control whether the base system for the builder (“localbase”) and the directory where the program is built are using tmpfs or not. Tmpfs is a memory-backed filesystem. If you disable one or both options, compilation times increase a lot because all the data that needs to be copied on setting up a builder and compiling software will be written to your disk. For an extreme example: On one of my machines, building and testing a port increased from slightly over 30 seconds with tmpfs to over 4 minutes (!) without it. Yes, that machine uses an HDD and it was occupied with other things besides building packages. But there is more than a negligible impact if you disable tmpfs.

So when to do it in the first place? If you’re on a machine with little RAM you might have to do this. Some ports like LLMV or Firefox require lots of RAM to build. If your system starts swapping heavily, disable tmpfs. Ideally build those ports separately and leave tmpfs on for all the rest.

Then there’s option M which toggles the fancy colored text UI on or off. If you turn it off, you’ll get a simple monochrome log-like output of which builder started or finished building which port. Every now and then the info that’s in the top bar of the UI (e.g. number of packages completed, number of packages remaining, skips, etc) gets printed.

Finally we have option N which toggles using pre-built packages on or off. It’s off by default which means that Synth will build everything required for the selected package set. If you enable this option and have e.g. the official FreeBSD repository enabled as well, it will try to fetch suitable packages from there that can be used as the buildtime or runtime dependencies of the packages that still need to be built. If you’re mixing official and custom packages this could be a huge time saver. I admit that I have never used this option.

And what’s this profile thing? Well, ok. The configuration we’ve been looking at is for the default profile. You can create additional ones if you want to. And that’s where the directories that I told you to ignore come into play. You could for example create a profile to use a different ports tree (e.g. the quaterly branch) and switch between the profiles to build two different packages sets on one machine. While Synth can do this, that is the point where I’d advise you to try out Poudriere instead. The main benefit of Synth over Poudriere is ease of use and when you’re trying to do clearly advanced things with Synth you might as well go for the officially supported FreeBSD package builder instead.

Compiler cache

Let’s disable the curses UI just to see what Synth looks like without it and save the config change. If you plan to build packages regularly on your system, you will definitely want to set up the compiler cache. To be able to use it, we first need another package installed, though: ccache. We’re going to build it and then install it manually this time:

# synth just-build devel/ccache
# pkg add /var/synth/live_packages/All/ccache-3.7.1_1.txz

Then we’re going to create a directory for it and go back to Synth’s configuration menu:

# mkdir -p /var/tmp/ccache/synth
# synth configure

Now change H to point to the newly created directory and save. Synth will use ccache now. But what does it do? Ccache caches the results from compilation processes. It also detects if the same compilation is happening again and can provide the cached result instead of actually compiling once again. This means that the first time you compile it doesn’t make a difference but after that the duration of building packages will drop significantly. The only cost of this is that the cached results take up a bit of space. Keep ccache disabled if drive space is your primary concern. In all other cases definitely turn it on!

Ccache directory and config after the system upgrade

Updating the system

Next is using Synth to update the system.

# synth upgrade-system

After determining which packages need to be built / rebuilt, Synth will start doing so. We’ve turned off the text UI, so now we get only the pretty simplistic output from standard text mode.

Package building in pure text mode

It provides you with the most important information about builders: Which builder started / finished building which package when. You don’t get the nice additional info about which state it’s in and how long it has been busy building so far. As mentioned above, Synth will print a status line every couple of minutes which holds the most important information. But that’s all.

Status lines amidst the builder output

What happens if something goes wrong? I simulated that by simply killing one of the processes associated with the rust builder. Synth reports failure to build rust and prints a status line that shows 17 package skips. In contrast to the curses UI it does not tell you explicitly which ones were skipped, though!

Simulated package build failure

When Synth is done building packages, it displays the tally as usual and does some repository cleanup by removing the old packages. Then it rebuilds the repository.

Tally displayed after completion and repository cleanup

Since we asked Synth to upgrade the system, it invokes pkg(8) to do its thing once the repository rebuild is complete.

Package repository rebuilt, starting system upgrade

And here’s why I strongly prefer prepare-system over upgrade-system: The upgrade is initiated whether there were failed packages or not. And since pkg(8) knows no mercy on currently installed programs when they block upgrades, it will happily remove them by default! To be safe it makes sense to always review what pkg(8) would do before actually letting it do it. Yes, it’s an additional step. Yes, most of the time you’ll be fine with letting Synth handle things. But it might also bite you. You’ve been warned.

System upgrade in progress – it may remove packages!

Every now and then you may want to run synth purge-distfiles (unless you have unlimited storage capacity, of course). It will make the tool scan the distinfo files of all ports and then look for distfile archives of obsolete program versions to remove.

Cleaning up old distfiles with purge-distfiles

There might not be a large gain in our example case, but things do add up. I’ve had Synth reclaim multiple GB on a desktop machine that I regularly upgraded by building custom packages. And that’s definitely worth it.

What’s next?

The next article will cover some leftover topics like logs, web report and repository sharing.

FreeBSD package building pt. 2: Basic Synth

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/freebsd_package_building_pt2.gmi

The previous article was an introduction into package building on FreeBSD in general. It also included references to various other articles about package management and working with ports. Familiarity with those topics is assumed here. If you don’t feel confident, have a look there and do a bit of reading first. Or just go ahead and do some research if you hit something that you have trouble understanding – package building is not a beginner’s topic but it ain’t rocket science, either.

In the following section I’m following a “naive approach” of just installing and exploring Synth on a typical desktop system. Afterwards I’m switching to the test system that was described in the previous post so that I can show updating, too.

Synth basics

So let’s get started with Synth now, shall we? The first thing to do is of course installing it:

# pkg install synth

If you want to or need to build it from source instead, the origin is ports-mgmt/synth. It has one compile-time option: Enabling or disabling the watchdog feature. By default it is enabled which means that Synth is supervising the build processes and if one looks like it has stalled, the watchdog will kill it. When you are building on very slow hardware (why are you doing this in the first place?) it may be an anti-feature that causes very complex ports (e.g. Firefox) to be killed even though it would otherwise eventually finish. In that case you can disable the watchdog option and rebuild Synth. Or you go to the junkyard and get some more capable hardware!

Synth port options

As with every new piece of software, it’s probably a good idea to get an overview of what it can do:

% synth help
Summary of command line options - see synth.1 man page for more details
===============================================================================
synth status              Dry-run: Shows what 'upgrade-system' would build
synth configure           Brings up interactive configuration menu
synth upgrade-system      Incremental rebuild of installed packages on system.
                          Afterwards, the local repository is rebuilt and the
                          system packages are automatically upgraded.
synth prepare-system      Like 'upgrade-system' but ends when repo is rebuilt
synth rebuild-repository  Rebuilds local Synth repository on command
synth purge-distfiles     Deletes obsolete source distribution files
synth status-everything   Dry-run: Shows what 'everything' would build
synth everything          Builds entire ports tree and rebuilds repository
synth version             Displays version, description and usage summary
synth help                Displays this screen
synth status [ports]      Dry-run: Shows what will be rebuilt with given list
synth build [ports]       Incrementally build ports based on given list, but
                          asks before updating repository and system
synth just-build [ports]  Like 'build', but skips post-build questions
synth install [ports]     Like 'build', but upgrades system without asking
synth force [ports]       Like 'build', but deletes existing packages first

synth test [ports]        Just builds with DEVELOPER=yes; pre-deletes pkgs

[ports] is a space-delimited list of origins, e.g. editors/joe editors/emacs.
It may also be a path to a file containing one origin per line.

That’s a nice list and even better: The commands are explained in a way that mere mortals can understand. Basically it can compare installed program versions to those in ports for you with synth status. You can configure synth with synth configure (we’ll look at this later, the default config is fine for now). And it can bulk-build packages and use those to upgrade the system for you with synth upgrade-system. And those are the important ones that you’ll need to begin with.

There’s also synth prepare-system in case you want Synth to only build the packages (maybe you want to do the upgrade manually later). With synth rebuild-repository you can make the tool, well, rebuild the repository from the currently built packages. Since it does this after finishing package builds, you will need it only if you cancelled building somewhere in the middle and want it to rebuild the repo anyway. The synth purge-distfiles command is useful if you use Synth for a while. It will scan for no longer needed distfiles (for obsolete program versions) and potentially free quite a bit of space for you.

And that’s it. The synth status-everything and synth everything commands are only useful if you want to build packages for the entire FreeBSD ports collection. But that’s certainly not basic usage! The various options that act on a single port or a list of ports is also advanced usage. They’ll come in handy if you plan on developing ports, build packages for other machines to use or have special needs. If you plan to keep your machine’s installed packages consistent for production use, know that these commands exist but stay away.

Package status

So much for some theory and on to actually doing something. Let’s ask Synth about the status of our packages:

# synth status
It seems that a blank PORTSDIR is defined in /etc/make.conf
Configuration failed to load.

Alright, this message can be a little confusing as by default there is no such file in FreeBSD! What Synth actually wants to tell you is that it cannot find a ports tree on the system and thus cannot do anything. So let’s get the ports in place right now:

# portsnap auto

With the ports tree available on our system we can try again:

# synth status
Configuration invalid: [C] Distfiles directory: /usr/ports/distfiles
Rather than manually creating a directory at this location, consider
using a location outside of the ports tree. Don't forget to set
'DISTDIR' to this new location in /etc/make.conf though.

This time the problem is pretty straight-forward: The distfiles directory does not exist. Synth gives us some advice in case we want to use a custom location, however I prefer the standard one.

# mkdir /usr/ports/distfiles
# synth status
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
 progress: 3.32%

Depending on your build machine this can take anything from a couple of minutes to a quarter of an hour or so. Before FreeBSD introduced port flavors, it was a matter of seconds, but for a few years now Synth has to do some additional work up-front. When it’s done it will print something like this:

Querying system about current package installations.
> Installed package ignored, devel/py-evdev package unmatched
> Installed package ignored, devel/py-pyudev package unmatched
> Installed package ignored, devel/py-six package unmatched
> Stand by, comparing installed packages against the ports tree.
> Stand by, building pkg(8) first ... done!
> These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
>   N => print/indexinfo
>   N => devel/gettext-runtime
> [...]

For the first run, Synth has to build all of the packages. Therefore all are marked new. But what’s that “Installed package ignored” thing? That usually happens if a package is installed on the system that was built from a port that does no longer exist in the new tree. Sometimes it’s also weirdness that can happen with flavored ports. Disregard this if it’s only affecting packages that you don’t want to use directly. The correct ones will be pulled in as dependencies anyway. Now let’s build our first package set:

# synth prepare-system

After gathering some information, Synth will start building the packages that are currently installed on this machine (which for simplicity’s sake resemble those of the test system). Let’s have a look:

Synth started building packages

By default, Synth shows a nice curses-based text UI that displays a lot of information (see screenshot above). It shows the total count of packages to build, the number of packages that remain to be built and the count of already built packages. In case a port does not successfully build on this system (update ports and try again – if the problem persists either file a bug report or contact the maintainer), Synth displays that, too. Ignored packages are those that don’t work on your particular system; maybe the application is known to not build on the version of FreeBSD you are using or not on your architecture (if for example you’re building on ARM). Skipped count goes up if a failed port was a dependency for others. Obviously Synth cannot build them.

Synth also displays the current system load: A load of e.g. 2.0 means that two cores of your CPU are completely busy. If your CPU supports Hyper-threading that basically doubles the available core count. A load higher than 8.0 on a 4 core system with HT means that the system has currently more work than it’s able to fulfill concurrently. It’s not ideal but it’s not something to worry too much about, either. Watch swap usage. A little bit of swapping for large ports is not much of a problem. But if your system swaps a lot that will slow down the package building a lot. Should you manage to run out of swap, package builds will fail. You can adjust Synth’s configuration if you’re unhappy with either load or swap usage. But we’ll get to that.

Synth after about 20 minutes

There’s also the number of packages built per hour and the impulse as well as the elapsed time. Initially packages per hour and impulse are identical because the two are similar. However the former is the average number of packages built over the whole build time while the latter is the number of packages built within the last 500 seconds.

But that’s only the top bar. The next part of the UI is for the builders. Builders are clean build environments (“chroot jails”) that are created before building a package and then teared down afterwards. The next package gets a new fresh environment. The number of builders means how many packages can be building concurrently. On the screenshot you can see 6 builders which are color-coded to be easier to distinguish. This may look different on your machine and here’s why: Synth tries to guess reasonable defaults for your machine. If you have an old dual-core PC, it will use less than the six builders it deemed right for the quad-core i7 that I used as my test machine here. Expect to see a much higher number of builders on modern servers with a higher core count.

Idle builders after 5 hours

For each builder you see how long it has been working already, which phase it is currently in (remember the various build targets of the ports infrastructure?) which port the builder is occupied with and how many lines of log output the builder has has produced so far. Especially the last information is not going to help you a lot when you begin building packages. After a while you’ll know roughly how many lines of output some of the big ports like LLVM produce and can judge if it’s going to take another two hours to finish or more like half an hour. (And then a new version of LLVM comes out which takes even longer to build so that your previous idea of “how many lines” is no longer valid. That’s how things go.)

And finally there’s the rest of the screen made up of a list of recently finished packages. If you take a look at the second screenshot, you’ll see some ports where the origin ends with @py37. Here the builder is busy building a flavored port – and it’s building for Python 3.7. What’s @native you ask? Well, Python ports are a typical example of flavored ports but they are not the only ones. The binutils port for example is able to be built as part of a native toolchain or a cross toolchain in case you want to e.g. cross compile packages for riscv on your much more powerful amd64 machine.

First builder has shut down

What’s the deal with idle builders like those on screenshot 4? Idle builders are the ones that Synth has already prepared for another package to build in but has not been able to use so far. Take a look at screenshot 4: There’s 53 more packages to build but only two builders are occupied with work while four are idle. Can you guess why? The two ports currently building are LLVM10 and Rust. And the “problem” is that all other 51 packages that are still on the list depend on either LLVM or Rust directly or indirectly! So while there’s more work for the builders, Synth has to wait for the dependencies to finish building before it can start building the remaining packages.

At some point no further builders will be required. In that case Synth shuts them down (see screenshot 5).

Last package build starting after about 6 hours

When Synth has finished building all the packages requested, it will present you with the final statistics. Then it cleans up the package directory, removing obsolete old packages. Eventually it will build a valid pkg(8) repository for you – and then it’s finally done.

Synth has completed its task

Should you ever want to quit a package run early, try to avoid CTRL-C. If you press CTRL-Q instead, Synth will shutdown gracefully. This means that now new builders will be started and the tool exits properly once those that are already running complete.

Pkg(8) repositories

I’m covering the case where you want to use your own package repository instead of the official FreeBSD one. If you want to use both, make sure you read what I pointed my readers to in the previous article. Then configure your new repository as I do here but simply don’t disable the official repo.

The standard FreeBSD repository is configured in /etc/pkg/FreeBSD.conf. Do not make changes there, though! This file belongs to the base system and upgrades may overwrite it. If you want to change the settings, create /usr/local/etc/pkg/repos/FreeBSD.conf (and the missing directories). The latter file will override the former. In case you just want to disable the official package repository, simply put this single line into the file:

FreeBSD: { enabled: no }

Synth should automatically generate a config file there if you use it to upgrade the system. The file is called /usr/local/etc/pkg/repos/00_synth.conf and has the following content:

# Automatically generated.

Synth: {
  url      : file:///var/synth/live_packages,
  priority : 0,
  enabled  : yes,
}

Now you only need to update the repository information:

# pkg update

And that’s it. Pkg(8) will now happily use your local package repository.

What’s next?

The next article will feature Synth configuration, upgrading and advanced usage.

FreeBSD package building pt. 1: Introduction and test system

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/freebsd_package_building_pt1.gmi

In 2017 I started to post a series of well-received articles that was meant to culminate in a tutorial on building packages on FreeBSD. I wrote about the history of package managers, about how to use FreeBSD’s pkg(8) package manager (part 1 and part 2) as well as an introduction to working with ports (part 1 and part 2).

Then I had to stop for technical reasons (after a major change to the ports infrastructure, the tool that I wanted to write about had not been updated to work with the new ports tree, yet!). In 2019 I eventually found the time to publish two more articles that were originally meant to come right after the 2017 posts. They covered using classic tools for ports management (part 1 and part 2).

They were meant to be the stepping stone to what I actually wanted to cover: Package building! The second article ended with:

I’ve planned for two more articles that will cover building from ports the modern way(tm) – and I hope that it will not take me another two years before I come to it…

That was… Yikes! In fall 2019… So I’d better hurry up now. I had hinted that using ports directly was not the recommended thing to do anymore and that while you should use packages unless you need ports, you really should use packages even in the latter case! And that’s what we’re going to do in the next few articles.

Why roll your own packages?

There are valid reasons to not use FreeBSD’s official packages:

  • Most frequently you need build-time options configured differently.
  • Or your organization uses a different default version of a scripting language (Perl, Python, Ruby, …) that what FreeBSD chose at the time being.
  • Maybe you’re running a customized version of FreeBSD that’s not officially supported.
  • Perhaps you need programs where the license forbids binary redistribution.
  • Or you use custom ports for some programs that of course cannot be available in official packages then.

Things like that. People choose ports when they need more control over their applications. There are good reasons to avoid using ports the traditional way, too, however:

  • It takes time and resources to build everything from source; especially if you do this on multiple machines, it’s a waste.
  • It makes updates much more complicated (see the second 2019 post mentioned above).
  • It clutters your system with build-time dependencies that need to be installed first so that the actual programs can be built.
  • Depending on what other software your machine has installed, the resulting programs might differ from that of other machines even if they built the same port!

While the former two points are mostly relevant if you manage multiple machines, I’d recommend rolling your own packages even for the single FreeBSD workstation that you use – if the official packages don’t suit you. Let me state this again: Try to go with the packages provided by the FreeBSD project first. Build your own packages if you have to (educating yourself is a completely valid reason to do it anyway, though).

Package builders

When you decide to roll your own packages, you have two options: Synth, the much easier and nicer package builder and Poudriere, the advanced build tool that FreeBSD uses and provides official documentation for.

Which one should you choose? I’m going to show how to work with both so you can compare them and make an informed decision. If you’re just getting started, you may want to give Synth a try. It is also a good choice when you use DragonFly BSD, too: The dsynth tool that they have in base was inspired by Synth (and if it wasn’t written in Ada they certainly would just have imported it instead of creating a re-implementation in C). You should also know that the Synth project is in maintenance mode. Its author still fixes bugs and takes pull requests on GitHub, but it’s feature-complete.

The main advantage of Synth is that it’s dead simple to setup and use, too, whereas Poudriere is a somewhat complex beast. Synth also shines when you want to use it to keep one machine up to date with packages as it can do those updates for you. Poudriere on the other hand allows you to do things like maintaining package repositories for multiple versions of FreeBSD as well as multiple architectures from one build machine. If you need that, forget Synth.

Ports and Git

One major change that was made in FreeBSD since the previous article was published is that the project migrated to using Git repositories. FreeBSD development started on CVS but in 2008 the src repository was successfully migrated to using Subversion. In mid 2012 docs and ports also made the switch. Subversion has been used ever since until December 2020 when doc and src transitioned to Git. After some remaining issues were solved, ports also migrated to Git in April 2021. While src changes get pushed back to Subversion for FreeBSD’s branches of 11 and 12, when it comes to ports, an era has ended.

Get rid of that habit of visiting svnweb.freebsd.org and start getting used to working with cgit.freebsd.org instead.

If you are unsure of your Git skills, you may want to at least skim over the Git-primer in FreeBSD’s documentation.

At least get familiar with the concepts. You should for example know that Git is able to do things like a shallow clone; looking things up when you need them is no problem but not being aware of them at all is.

While both Subversion and Git are used for version control and both use repositories, they are fundamentally different. Subversion is the best known version control system of the second generation (so-called (centralized) “networked VCS”). Git is the most popular one of the third generation (“decentralized VCS”).

If you use Subversion, there’s a central repository somewhere and you checkout a certain revision of the versioned files from there. The files that were checked out form your working directory. In Git you clone a remote repository and thus receive all the data needed to get a local copy of the whole repo. The files for the working directory are checked out from the local copy. The major difference is: You have the whole history available locally.

In case you have no need for all that data, you do a shallow clone instead of a regular full clone. To give you an example: If you do a shallow clone of the current ports tree today, the result is about 840 MB in /usr/ports – of which 85 MB in size is the Git repository. A full clone is about 1.7 GB in size of which about 920 MB is for the repo! So if you don’t need the history, save some space on your drive and save some donated bandwidth of the FreeBSD project.

Considerations before you start

While you can certainly start rolling your own packages on a fresh system, it’s also fine to begin doing so on a system that has used the standard FreeBSD ports or packages so far. There’s nothing wrong with that actually. Taking the opposite way and going back from custom packages to the official ones is also possible, of course. That latter case requires some more planning, though. Think about why you started building your own packages in the first place, find out if the standard packages fit your current needs and think about the consequences. If you’re uncertain, you may want to start over with regular packages. If you want to make the switch back, it’s best to re-install all the packages (use the flag -f with pkg upgrade). If you’re using ZFS in a way that supports Boot Environments, create one first.

In fact you can even choose to use both official and custom packages on the same machine: Instead of building the full set of packages that you need, you build just the packages yourself that you need to customize and use the packages from the standard FreeBSD package repositories for the rest. This works by configuring your package repository as an additional one for pkg(8) and giving it a higher priority. If you consider doing this and using multiple package repositories, be sure to

% man 5 pkg.conf

first. It doesn’t hurt to skim over the whole manpage, but make sure that you at least read the section REPOSITORY CONFIGURATION. I used that approach on a weaker machine in the past, but wouldn’t generally recommend it. If your hardware is recent enough, you can compile everything yourself. Otherwise it might make more sense to build on a somewhat beefy system and distribute the packages to your other machine(s). But that’s only me. Maybe mixing packages is the right solution for your use case.

Building a test system

The rest of this post is about building a test system to simulate an environment with pre-installed packages. I do this so that I can show off a few things regarding updates in the next post. If you want to follow along, make sure that you have Git installed and that your directory or dataset /usr/ports as well as /usr/src is empty. I’m assuming that you’re using FreeBSD 13.0.

The first thing to do is getting the ports tree in place. I’m doing a shallow clone of the 2021 first quarter branch (so I can update later). Then I clone operating system source repository as well:

# git clone --depth 1 -b 2021Q1 https://git.freebsd.org/ports.git /usr/ports
# git clone --depth 1 -b releng/13.0 https://git.freebsd.org/src.git /usr/src

Now we don’t need the packages anymore and forcefully remove them all (including pkg). The reason for this is that we want to build the older versions of the old ports tree that we just cloned.

# pkg delete -af

For convenience we’re going to build all the required programs using portmaster as discussed in a previous article (see the beginning of this post). Of course we need to build and install it first:

# make -C /usr/ports/ports-mgmt/portmaster install clean

Alright. Now we’re building some leaf ports to simulate a very simple development system. These ports draw in a number of dependencies so that we end up with about 350 packages (which is still a very low count for a desktop system). I’m building tmux and sudo simply because I always need to have them available and git just because we needed it before anyway. The rest is the graphics drivers, a simple subset of X11 + the command to set a different keyboard layout as well as the awesome window manager and the simple but nice GTK+ terminal emulator called sakura. Since I’ll need to take screenshots for the upcoming posts, I figured that I might include an application for that, too.

# portmaster -dG sysutils/tmux security/sudo devel/git graphics/drm-kmod x11/xorg-minimal x11/setxkbmap x11-wm/awesome x11/sakura x11/xfce4-screenshooter-plugin

And that’s it for today.

What’s next?

Next station is taking a look at building packages and updating the system using Synth.

Dystopian Open Source

[New to Gemini? Have a look at my Gemini FAQ.]

This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/dystopian_open_source.gmi

Happy New Year dear reader! The other day I watched a video on YouTube that had only 6 views since last October. It is about a very important topic, though, and I wish it would have a larger impact as well as get more people alarmed and thinking about the current trends in Open Source. This is not a “OMG we’re all doomed!!1” post, but I want to talk about what I feel are grave dangers that we should really, really aim some serious consideration at.

“Pay to Play”

For the readers who would like to watch the video (about 7 minutes), it’s here. Some background info: It’s by Lucas Holt. He is the lead developer of MidnightBSD, a project that began as a Fork of FreeBSD 6.1 and aimed for better usability on the desktop. There were a couple of people who contributed to the project over time, but it never really took off. Therefore it has continued as a project almost entirely done by one man.

It’s not hard to imagine just how much work it is to keep an entire operating system going; much larger teams have failed to deliver something useful after all. So while it’s no wonder that MidnightBSD is not in a state where anybody would recommend it to put to everyday usage, I cannot deny that I admire all the work that has been done.

Holt has merged changes back from FreeBSD several times, eventually updating the system to basically 11.4 plus the MidnightBSD additions and changes. He maintains almost 5,000 ports for his platform (of course not all are in perfect shape, though). And he has kept the project going since about 2006 – despite all the taunting and acid-tongued comments on “the most useless OS ever” and things like that. Even though I never found somewhat serious use for MidnightBSD (and I tried a couple of times!), considering all of that he has earned my deepest respect.

To sum up the video: He talks about a trend in Open Source that some very important projects started to raise the bar on contributing to them. Sometimes you’re required to employ two full-time (!) developers to be considered even worth hearing. Others require you to provide them with e.g. a paid Amazon EC2 instance to run their CI on. And even where that’s not the case, some decision makers will just turn you down if you dare to hand in patches for a platform that’s not a huge player itself.

Quite a few people do not even try to hide that they only ever care about Linux and Holt has made the observation that some of the worst-behaving, most arrogant of these are – Redhat employees. There are people on various developer teams that choose to deliberately ruin things for smaller projects, which is certainly not good and shouldn’t be what Open Source is about.

What does Open Source mean to us?

At a bare minimum, Open Source only means that the source for some application, collection of software or even entire operating system is available to look at. I could write some program, put the code under an extremely restrictive license and still call this thing “Open Source” as long as I make the code available by some means. One could argue that in the truest sense of the two words that make up the term, that would be a valid way to do things. But that’s not what Open Source is or ever was about!

There are various licenses out there that are closely related to Open Source. Taking a closer look at them is one great way to find the very essence of what Open Source actually is. There are two important families of such licenses: The so-called Copyleft licenses and the permissive licenses. One could say that downright religious wars have been waged about which side holds the one real truth…

People who have been reading my blog for a while know that I do have a preference and made quite clear which camp I belong to, even though I reject the insane hostility that some zealots preach. But while the long-standing… err… let’s say: controversy, is an important part of Open Source culture, the details are less relevant to our topic here. They basically disagree on the question of what requirements to put in the license. Should there be any at all? Is it sufficient to ask for giving credit to the original authors? Or should users be forced to keep the source open for example?

Both license families however do not dispute the fundamental rights given to users: They want you to be able to study the code, to build it yourself, to make changes and to put the resulting programs to good use. While it’s usually not explicit, the very idea behind all of Open Source is to allow for collaboration.

Forkability of Open Source projects

Over the years we’ve seen a lot of uproar in the community when the leaders of some project made decisions that go against these core values of Open Source. While some even committed the ultimate sin of closing down formerly open code, most of the time it’s been slightly less harsh. Still we have seen XFree86 basically falling into oblivion after Xorg was forked from it. The reason this happened was a license change: One individual felt that it was time for a little bit of extra fame – and eventually he ended up blowing his work to pieces. Other examples are pfSense and OPNsense, Owncloud and Nextcloud or Bacula and Bareos. When greed strikes, some previously sane people begin to think that it’s a good idea to implement restrictions, rip off the community and go “premium”.

One of the great virtues of Open Source is that a continuation of the software in the old way of the project is possible. With OPNsense we still have a great, permissively licensed firewall OS based on FreeBSD and Pf despite NetGate’s efforts to mess with pfSense. Bareos still has the features that Bacula cut out (!) of the Open Source version and moved to the commercial one. And so on. The very nature of Open Source also allows for people to pick up and continue some software when the original project shuts down for whatever reason.

There are a lot of benefits to Open Source over Closed Source models. But is it really immune to each and every attack you can aim at it?

Three dangers to Open Source!

There is always the pretty obvious danger of closing down source code if the license does not prohibit that. Though I make the claim that this in fact mostly a non-issue. There are a lot of voices out there who are going hysteric about this. But despite what they try to make things look, it is impossible to close down source code that is under an Open Source license! A project can stop releasing the source for newer versions, effectively stopping to distribute current code. But then the Open Source community can always stop using that stuff and continue on with the a fork that stays open.

But we haven’t talked about three other immanent dangers: narrow-mindedness, non-portability and leadership driven by monetary interest.

Narrow-mindedness

One could say that today Open Source is victim of its overwhelming success. A lot of companies and individual developers jumped the wagon because it’s very much beneficial for them. “Let’s put the source on GitHub and people might report issues or even open pull-requests, actively improving our code – all for free!” While this is a pretty smart thing to do from a commercial point of view, in this case software code was not opened up because somebody really believes in the ideas of Open Source. It was merely done to benefit from some of the most obvious advantages.

Depending on how far-sighted such an actor is, he might understand the indirect advantages to the project when keeping things as open as possible – or maybe not. For example a developer might decide that he’ll only ever use Ubuntu. Somebody reports a problem with Arch Linux: Close (“not supported!”). Another person opens a PR adding NetBSD support: Close (“Get lost, freak!”).

Such behavior is about as stupid and when it comes to the values also as anti Open Source as it gets. Witnessing something like this makes people who actually care about Open Source cringe. How can anybody be too blind to see that they are hurting themselves in the long run? But it happens time and time again. By turning down the Arch guy, the project has probably lost a future contributor – and maybe the issue reported was due to incompatibilities with the never GCC in Arch that will eventually land in Ubuntu, too, and could have been fixed ahead of time…

Open Source is about being open-minded. Just publishing the source and fishing for free contributions while living the ways of a closed-source spirit is in fact a real threat to Open Source. I wish more people would just say no to projects that regularly say “no” to others (without a good reason). It’s perfectly fine that some project cannot guarantee their software to even compile on illumos all the time. But the illumos people will take care of that and probably submit patches if needed. But refusing to even talk about possible support for that platform is very bad style and does not fit well with the ideals of Open Source.

If I witness that an arrogant developer insults, say a Haiku person, I’ll go looking for more welcoming alternatives (and am perfectly willing to accept something that is technically less ideal for now). Not because I’ve ever used Haiku or do plan to do so. But simply because I believe in Open Source and in fact have a heart for the cool smaller projects that are doing interesting things aside of the often somewhat boring mainstream.

Non-portability

Somewhat related to the point above is (deliberate) non-portability. A great example of this is Systemd. Yes, there have been many, many hateful comments about it and there are people who have stated that they really hope the main developer will keep the promise to never make it portable “so that *BSD is never going to be infected”.

But whatever your stance on this particular case is – there is an important fact: As soon as any such non-portable Open Source project gains a certain popularity, it will begin to poison other projects, too. Some developers will add dependencies to such non-portable software and thus make their own software unusable on other platforms even though that very software alone would work perfectly fine! Sometimes this happens because developers make the false assumption that “everybody uses Systemd today, anyway”, sometimes because they use it themselves and don’t realize the implication of making it a mandatory requirement.

If this happens to a project that basically has three users world-wide, it’s a pitty but does not have a major impact. If it’s a software however that is a critical component in various downstream projects it can potentially affect millions of users. The right thing here is not to break solidarity with other platforms. Even if the primary platform for your project is Linux, never ever go as far as adding a hard dependency on Systemd and other such software! If you can, it’s much better to make support optional so that people who want to use it benefit from existing support. But don’t ruin the day for everybody else!

And think again about the exemplary NetBSD pull-request mentioned above: Assume that the developer had shown less hostility and accepted the PR (with no promises to ever test if things actually work properly or at all). The software would have landed in Pkgsrc and somebody else would soon have hit a problem due to a corner case on NetBSD/SPARC64. A closer inspection of that would have revealed a serious bug that remained undetected and unfixed. After a new feature was added not much later, the bug became exploitable. Eventually the project gained a “nice” new CVE of severity 9.2 – which could well have been avoided in an alternate reality where the project leader had had a more friendly and open-minded personality…

Taking portability very seriously is exceptionally hard work. But remember: Nobody is asking you to support all the hardware you probably don’t even have or all the operating systems that you don’t know your way around on. But just be open to enthusiasts who care for such platforms and let them at least contribute.

Leadership with commercial interests

This one is a no-brainer – but unfortunately one that we can see happening more and more often. Over the last few years people started to complain about e.g. Linux being “hi-jacked by corporations”. And there is some truth to it: There is a lot of paid work being done on various Open Source projects. Some of the companies that pay developers do so because they have an interest in improving Open Source software they use. A couple even fund such projects because they feel giving back something after receiving for free is the right thing to do. But then there’s the other type, too: Corporations that have their very own agenda and leverage the fact that decision makers on some projects are their employees to influence development.

Be it the person responsible for a certain kernel subsystem turning down good patches that would be beneficial for a lot of people for seemingly no good reason – but in fact because they were handed in by a competitor because his employer is secretly working on something similar and has an interest to get that one in instead. Be it because the employer thinks that the developer is not payed to do anything for platforms that are not of interest to its own commercial plan and is expected to simply turn those down to “save time” for “important work”. Things like that actually happen and have been happening for a while now.

Limiting the influence of commercial companies is a topic on its own. IMO more projects should think about governance models much more deeply and consider the possible impacts of what can happen if a malicious actor buys in.

Towards a more far-sighted, “vrij” Open Source?

As noted above, I feel that some actors in Open Source are too much focused on their own use-case only and are completely ignorant of what other people might be interested in. But as this post’s topic was a very negative one, I’d like to end it more positively. Despite the relatively rare but very unfortunate misbehaving of some representatives of important projects, the overwhelming majority of people in Open Source are happy to allow contributions from more “exotic” projects.

But what’s that funny looking word doing there in the heading? Let me explain. We already have FOSS, an acronym for “Free and Open Source Software”. There’s a group of people arguing that we should rather focus on what they call FLOSS, “Free and Libre Open Source Software”. The “libre” in there is meant to put focus on some copyleft ideas of freedom – “free” was already taken and has the problem that the English word doesn’t distinguish between free “as in freedom” and free of charge. I feel that a term that emphasizes the community aspect of Open Source, the invitation to just about anybody to collaborate and Open Source solidarity with systems other than what I use, could be helpful. How about VOSS? I think it’s better than fitting in another letter there.

Vrij is the Dutch word for free. Why Dutch? For one part to honor the work that has been done at the Vrije Universiteit of Amsterdam (for readers who noticed the additional “e”: That’s due to inflection). Just think of the nowadays often overlooked work of Professor Tanenbaum e.g. with Minix (which inspired Linux among other things). The other thing is that it’s relatively easy to pronounce for people who speak English. It’s not completely similar but relatively close to the English “fray”. And if you’re looking for the noun, there’s both vrijheid and vrijdom. I think the latter is less common, but again: It’s much closer to English “freedom” and thus probably much more practical.

So… I really care for vrij(e) Open Source Software! Do you?

FreeBSD on SPARC64 (is dead)

This was originally my January article that I decided to postpone for another one. So it’s really the February one – except for myself not getting around to publishing it before August due to… well, the world going crazy. So this is the January state of affairs for the most part.

On the plus side of all the havoc: I went over the article again and improved it. I intend to publish the other articles that I missed out one by one.

After getting my hands on an old SPARC64-based SUN machine, I’ve written a couple of posts about the actual hardware, installing and running OpenBSD as well as an illumos distro on it. I’ve also hinted that I wanted to try out my favorite OS on it, too: FreeBSD.

FreeBSD on SPARC64?

I’m coming pretty late to the party, because SPARC64 support in FreeBSD is apparently doomed: After the POWER platform made the switch to a LLVM/Clang-based toolchain, SPARC64 is one of the last ones that still uses the ancient GCC 4.2-based toolchain that the project wants to finally get rid off (it has already happened as I was writing this – looks like the firm plan was not so firm after all, since they killed it off early). And compared to the other platforms it has seen not too much love in recent times… SPARC64 being a great platform, I’d be quite sad to see it go. But before that happens let’s see what the current status is and what would need to be done if it were to survive, shall we?

The FreeBSD project provides ISO images for various platforms. At the time of my first attempt I downloaded a 12.0-RELEASE SPARC64 iso. Since my machine cannot boot from USB I can’t use the virtual CD drive that I typically use. So I went old-school and burned a CD. Then I prepared everything and turned the machine on. It started booting but before even reaching the installer or anything, I witness something completely unexpected – here’s the last few lines from the serial console:

nexus0:  type unknown (no driver attached)
rtc0:  at port 0x70-0x71 pnpid PNP0b00 on isa0
rtc0: registered as a time-of-day clock, resolution 1.000000s
uart0: console (9600,n,8,1)> at port 0x3f8-0x3ff irq 43 pnpid PNP0501 on isa0
uart1:  at port 0x2e8-0x2ef irq 43 pnpid PNP0501 on isa0
Timecounter "tick" frequency 548000000 Hz quality 1000
Event timer "tick" frequency 548000000 Hz quality 1000
Timecounters tick every 1.000 msec
Power Failure Detected: Shutting down NOW.

Wow, what’s that? FreeBSD thinks that the machine has a damaged PSU and immediately shuts down, giving me no chance to actually do anything. Now that was less than impressive… This was when I decided to try out OpenBSD instead. That worked well and the machine ran for a couple of days, compiling OpenBSD. So the power supply is definitely fine.

I did a little research and it seems that some models fire an interrupt that means “PSU is damaged” with other models. OpenBSD seems to know this for my machine and ignores it, but FreeBSD doesn’t.

Rolling my own ISO

Ok, so what to do now? We have a defunct OS but we also have a string to look for. It’s not hard to find it in /sys/sparc64/pci/psycho.c. I don’t know C and thus cannot implement new features or do real debugging of any sort. But I can comment stuff out. And since this is a false alarm that should be sufficient. So I get rid of the check that ruins the boot for me and save.

Now, one of the things that I love about FreeBSD is its build system. It’s so easy to build the OS – or to even cross-build it! It’s as simple as doing this:

cd /usr/src
make TARGET=sparc64 TARGET_ARCH=sparc64 buildworld buildkernel

This cross-builds FreeBSD for SPARC64. But how to get it onto my SUN machine? Fortunately there is a pretty interesting directory in /usr/src/release. The Makefile and scripts there allow for extremely easy creation of installation media. So it should just be two more commands and I’d end up with an ISO image that contains my modified FreeBSD. Let’s give it a try:

cd /usr/src/release
make cdrom NOPKG=1 NOPORTS=1 NOSRC=1 NODOC=1 TARGET=sparc64 TARGET_ARCH=sparc64

This means: Build a minimal install CDROM image, leaving out any packages, the ports tree as well as the source and documentation distribution sets. But what is this?

gpart: scheme 'VTOC8': Invalid argument
*** Error code 1

The manpage for gpart reveals the cause of the problem: By default VTOC8 is not supported on amd64 kernels (which is a perfectly reasonable choice). However solving the problem is as easy as adding GEOM_PART_VTOC8 to the kernel options and building a custom kernel on my amd64 build machine.

If you’ve never worked with a custom kernel (as it’s far less often required than it was in the past), have a look in /usr/src/sys/amd64/conf. You’ll find a file called GENERIC there. It holds all the definitions for the standard kernel. While you could just edit this file it’s bad practice to do so. At least copy GENERIC to a new file:

cp /usr/src/sys/amd64/conf/GENERIC /usr/src/sys/amd64/conf/GENERIC-VTOC

Now let’s change the ident line to GENERIC-VTOC, too and add the option GEOM_PART_VTOC8 as the gpart manpage suggests. That’s all, now I can rebuild the kernel with

make -C /usr/src kernel KERNCONF=GENERIC-VTOC

This command line builds and installs the new kernel for me, I just have to reboot afterwards. And not too much later I have an ISO to burn that would finally let me install FreeBSD on the machine!

Building software

The first surprise for me: The installer is missing the option to install on ZFS as I have become used to. Looks like ZFS can be used, but the booting facility for sparc64 does not support booting from ZFS.

There are no pre-built packages available for sparc64. There used to be up until about a year ago, but now it’s building everything from ports.

After building tmux I decided to upgrade the system. Of course there’s no way to use freebsd-update, so it’s also compiling from source. I give -CURRENT a try, but the old GCC 4.2 in base cannot build that. Eventually I settle on 12-STABLE. Compiling it takes many hours, but it works without problems.

After rebooting, however, my system doesn’t come back up. Uh-oh. What’s happening here? I connect via the serial console – and see that I forgot to patch psycho.c… Fortunately I can just boot kernel.old and be good again. While editing the file again, I notice that I did unnecessary work before: There’s a sysctl that can turn the automatic shutdown off! So, yes, it’s just a simple hw.psycho.powerfail=0 in the loader.conf and the unmodified stable kernel boots. If I had entered set hw.psycho.powerfail=0 at the OK prompt in the loader, I wouldn’t even had to roll my own ISO. But I like exploring how FreeBSD works, so that extra step was not so bad after all.

Now having a pretty fresh system, I go back to exploring ports again – and run into problems. The default version of GCC on FreeBSD is currently 9.2, but that version is broken on sparc64. GCC8, GCC7 and GCC48 work and can be used to compile software (the latter needs the port edited, though, as sparc64 is not listed as a supported platform even though the port is fine). GCC6 is also broken after the latest minor release.

This is pretty unfortunate since it means that most software that requires a newer GCC to compile cannot be built easily. Sure, I could mess with the ports infrastructure to make it use GCC8 instead, but I’d rather investigate something else.

External GCC

While the PPC platform recently moved to the LLVM/Clang-based toolchain, SPARC64 is still stuck with the ancient GCC in base as mentioned before. For such projects, FreeBSD has created a path to move forward: Using an external toolchain as the base compiler.

Have a look in /usr/ports/base. You’ll find the two ports binutils and gcc6 there. This wiki page gives an overview of how many steps the various platforms have already made.

So I built the binutils and gcc packages, installed them – and broke my system. As I found out, the base ports actually replace said component in base. So after installing, it’s no longer binutils 2.17.50, but the much more recent 2.33.1 and gcc 6.5.0 instead of 4.2.1. The latter however is broken… So time to make installworld from my -STABLE source again to fix my system.

What to do next? Well, -CURRENT is where the exciting stuff happens, so I’ve got to get that onto my machine somehow! The native way is blocked, as I have no idea now to fix the compiler. But what about cross-compiling for now?

Cross-building -CURRENT

There’s a package with the name sparc64-xtoolchain-gcc available for my amd64 machine, so installing the cross-toolchain is about as easy as it gets. Let’s try to build FreeBSD with it!

It takes a while but of course finishes much more quickly compared to a native build on the old SUN hardware. Ok, next step is preparing a release for it, so that I can try it out. Unfortunately trying to do so leads to this:

--- libc.a ---
/usr/local/sparc64-unknown-freebsd13.0/bin/ranlib -D libc.a
--- libc.so.7.full ---   
collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
compilation terminated.
/usr/local/bin/sparc64-unknown-freebsd13.0-ld: BFD (GNU Binutils) 2.31.1 assertion fail elfxx-sparc.c:765
*** [libc.so.7.full] Error code 1

Ouch! What is this? It looks like the linker is broken! Time to fix it, eh? Except when you’re not a C programmer, what do you do now? After a moment of consideration I decided that this might be well beyond my skills – but sometimes you’re lucky. What if the change that broke it was rather simple to understand? You don’t know until you try, so here we go…

Linker games

There must have been a time where things worked, so I start by testing a build with an older version of Binutils: Using 2.28 yields a successful build with GCC 9 (gtest disabled). Same thing for 2.30. Using 2.31 however leads to the known linker problem. Aha, so the error was introduced during the development of Binutils 2.31. That’s good to know.

So now it was time to clone the git repo (I don’t quite understand why they put binutils and gdb into the same one but never mind) and dig into it. It took me a while to figure out exactly which commit broke things but was eventually able to find the culprit. It was a not a simple commit as I had hoped. Cleanup of old cruft had been done (like phasing out of a.out and COFF support) und that somehow also broke ELF. I couldn’t see why and so I started taking the commit apart.

Finally I traced the problem to a change in gas (the GNU assembler) where the cleanup lead to an oversimplification with targets that made gas on FreeBSD behave as if it was NetBSD! I provided the patch on the FreeBSD bugtracker (here) and it was accepted. I posted my patch to the upstream mailing list for Binutils, too (here) and a couple of days later Alan Modra who did the change back then got back to me after having merged a fix.

Once the patch hit the ports tree I was able to cross-compile a -CURRENT SPARC64 kernel on AMD64 out of the box without any need for patching. Yay!

Playing with cross-built kernels

Trying to boot up the kernel resulted in a panic quite early in the boot, but after getting this far I didn’t want to just give up now. So I filed another bug report and the problem was in fact solved really quickly. With that fix the kernel was able to boot further – only to hit another panic. Fortunately the second one was sorted out almost instantly as well on the same bug report.

Having a working -CURRENT kernel booted up felt just great. But what about the userland? It was also broken in multiple ways and did not compile. While it took me days to figure out all the breaking commits, reverting or otherwise fixing them for testing purposes, I ended up with a patched userland that did successfully cross-compile. The downside of things was that every single resulting binary that I tried out would just crash… At that point I hit a dead end. I submitted one final bug report (here) but as nobody had any suggestions for me this was the game over point.

Goodbye SPARC64!

When I learned that FreeBSD was about to drop SPARC64 support I hoped that I could step in and help in keeping it alive. This proved to be an illusion as obviously nobody with a deeper technical understanding than me had any real interest in that platform. I admit being deeply disappointed in January when SPARC64 was dropped. Not because it was dropped, but mainly because it was dropped early. Maybe a bit more time would have helped to keep it in, maybe it wouldn’t. We’ll never know.

But it’s probably a good thing that I didn’t get around to publish this article until months after the events. I’d like to especially thank Ed Maste, Mark Johnston and Baptiste Daroussin who merged fixes for a most obviously doomed platform as well as other people from the team who were helpful about it until the end. FreeBSD on SPARC64 failed to survive until 13.0-RELEASE, but the efforts that were made show how much of a professionally developed OS FreeBSD is.

So let’s put SPARC64 to rest in dignity. The team has stated that support over the lifespan of 11-STABLE and 12-STABLE will be done as “best effort” work since it’s gone in HEAD. Especially 12 will live for some time and I’ll try to continue doing things with my SunFire until the curtain falls some time after 12.x.

Should you abandon Linux and switch to *BSD?

The popularity of Linux skyrockets these days: More and more companies adapt it and even the just-above-average who doesn’t accept the imposition that is called “Windows 10” is often open to try it out. However at the same time, the popularity of said System seems to be fading among some of the more technical people, operating system enthusiasts and even followers of the FLOSS ideology.

Just recently an article called Why you should migrate everything from Linux to BSD was published and it has caught some attention and even replies like this one (it makes false claims like ZFS not being available on NetBSD and such, though.)

There have always been some posts like this, this is nothing too special. What I think is new, however, is the frequency that you can find discussions like this. Also the general tone has changed. Just a couple of years ago most Linux users wouldn’t even have bothered to comment such a thing. Today they seem to be much more open to learning about alternatives or in fact looking for something better than what they have. So what’s wrong with Linux?

It’s not about Systemd (alone)…

There are a lot of perfectly valid technical reasons to not want to use Systemd on your Linux system. But none of those could ever be an excuse for the hatred that this project has attracted. However there is a pretty simple explanation for that phenomenon: It’s how somebody is acting all high and mighty, simply dismissing valid critics and being a great example for a person with an arrogant attitude.

We have a lot of… let’s say… difficult persons it tech. Sometimes you think they have no manners, are being jerks or greatly overestimate their knowledge in certain fields. That’s ok and in fact they often are somewhat brilliant in a certain area. Most of us have learned to live with that.

And then there are people who think that they can dictate the one way to go. Well, there are project leaders who actually can do that due to being widely respected. But sometimes it’s different. Now when somebody wants to take something away from you (or he really doesn’t but you get the impression), you are likely to stand upon your defense.

Now when all of that unites in one person, you have the perfect boogeyman. Then all the technical aspects lose in weight and the feeling takes over. Which is not to say that feeling is not an important factor: If you don’t feel comfortable with Linux anymore, it might be time to move on.

The GNOME factor

The GNOME desktop is well known among *nix desktop users. It suits the needs some people but not others. That’s fine and there would not be any problem at all. However GNOME has a certain reputation which is not that nice… Why? Not because they didn’t accept some feature requests. Also not because are being ignoramuses when it comes to systems that are not so mainstream. No. They are notorious for cutting features that have already existed! This is what makes a lot of people mad.

I was a GNOME user myself and remember pretty well how I liked the file manager. To my dismay they removed so many features which I needed that the application became useless for me. I didn’t want to go looking for something else, but I was eventually forced to as the situation became unbearable. As I’m more of a calm guy, I didn’t go off at insult anybody, but other people did. And things got worse…

Today I’m a former GNOME user. This is *nix and not Windows. Nobody can force tiles upon us against our will. Yes, some projects think they know better than their users and that leads to the latter becoming upset. But as long as there are alternatives, we can move on.

Clumsy leaders

Recently Linus Torvalds spoke out against ZFS. Being the Linux “inventor” he has earned a lot of respect among the *nix community. I also used to hold him in high esteem despite his often ignoble behavior. However over the course of the last few years, I’ve lost a lot of respect for him.

The ZFS statement was the last coffin nail for me. He says that he always found ZFS more of a “buzzword” than anything else and that “benchmarks” didn’t make it look that good anyway. This is so far off the shot that I’m ashamed to ever have considered him a technical genius. He obviously does have no idea at all what problem ZFS actually solves! Speed benchmarks are all nice and well, but they are not why people want ZFS! And of course it’s far from being a buzzword – if you have valuable data today, you almost certainly want to bet on ZFS.

But not only is he ok with judging what he has not even bothered to take a look at from closer that several hundred meters – he’s also making completely stupid claims that make him look like a terribly ridiculous figure. According to Mr. Torvalds, ZFS had “no real maintenance behind it either any more”.

Ouch! He hasn’t even heard about OpenZFS, I’d guess. If you’re not in a closed-Solaris environment, this is what people are referring to when they say “ZFS”. Nobody outside the small, isolated isle of Oracle has any interest in ClosedZFS. Yes, Oracle laid off most of their Solaris staff and nobody knows if there is any noteworthy future for that OS. But not too long Solaris 11.4 was released – so even if Linus referred to the situation at Oracle, he’s not exactly right. In the case of OpenZFS, however, he could not be more wrong.

The OpenZFS is as alive as it could be. There are regular leadership meetings, many new features are being developed – and just recently a common repository for both Linux and FreeBSD was created, with other operating systems expected to join in! This is a moment of glory for collaboration in Open Source, but Linus didn’t hear a thing – or did he not want to hear anything? The fact that the second-in-command, GKH, has attacked ZFS about a year ago in a pretty questionable way, too, does not bode well.

Why do people leave Linux?

There have always been compelling technical reasons why you would choose *BSD over Linux (e.g. the complete operating system approach, much more consistent design, etc.). But I’d say that lately the the feeling part of it became much more important.

I left Linux because I was so sick and tired of the stupid fights between the hardcore fans of one distro or another and the unbearable arrogance of many. Yes, I also had the feeling that Linux was heading down the wrong way, too. I simply was no longer really happy with it and ready to try something new. There was a learning curve for sure, but the FreeBSD community is extremely friendly and while there are of course also people getting into disputes, I got the feeling that I described as “BSD is for grown-ups”. Not saying that there aren’t any really bright people in the Linux community, but on average I feel that the BSD users are more technical.

Others have stated similar reasons. The primary developer of ClonOS (that strives to be for FreeBSD what Proxmox is for Linux) wrote this:

According to the authors of the project, Linux is no longer a member of the common people, it is fully controlled by big commercial organization. while FreeBSD is developed mostly by enthusiasts. Today, Linux – it is a commercial machine for making money – is that it was Microsoft Windows in 90 years. While many Linux users have struggled against the Windows monopoly (CBSD author of one of them).

Yes, FreeBSD very far behind in their characteristics in comparing to Linux. Just look at the abundance of such powerfull decisions as the OpenVZ, Docker, Rancher, Kubernetis, LXD, Ceph, GlusterFS, OpenNebula, OpenStack, Proxmox, ISPPanel and a dozen others. All this is created by commercial companies for Linux and this is done very well. However, Linux is oversaturated with similar solutions. Therefore, it’s much more interesting to create it on FreeBSD, where nothing like that exists. This is an excellent challenge to improve and fix in FreeBSD.

We all love independence and freedom and FreeBSD today – an independent and free operating system, which is in the hands of ordinary people.

They are not alone. Even convinced followers of the FSF ideas have come to the conclusion that Linux may not be the right platform for them anymore. The people behind Hyperbola GNU/Linux have announced this:

Due to the Linux kernel rapidly proceeding down an unstable path, we are planning on implementing a completely new OS derived from several BSD implementations. This was not an easy decision to make, but we wish to use our time and resources to create a viable alternative to the current operating system trends which are actively seeking to undermine user choice and freedom.

This will not be a “distro”, but a hard fork of the OpenBSD kernel and userspace including new code written under GPLv3 and LGPLv3 to replace GPL-incompatible parts and non-free ones.

Reasons for this include:

Linux kernel forcing adaption of DRM, including HDCP.
Linux kernel proposed usage of Rust (which contains freedom flaws and a centralized code repository that is more prone to cyber attack and generally requires internet access to use.)
Linux kernel being written without security and in mind. (KSPP is basically a dead project and Grsec is no longer free software)
Many GNU userspace and core utils are all forcing adaption of features without build time options to disable them. E.g. (PulseAudio / SystemD / Rust / Java as forced dependencies)

As such, we will continue to support the Milky Way branch until 2022 when our legacy Linux-libre kernel reaches End of Life.

Will *BSD be a better OS for you?

So the big question is: If you are a Linux user, should you make the switch, too? I won’t unconditionally say yes. It really depends.

Are you happy with the overall situation in Linux? In that case there’s no need to migrate anything over. However you might still want to give a BSD of your choice a try. Perhaps you find something that you like even better? If you spend a bit of time exploring a BSD, you will find that several problems can be solved in other ways than those you are familiar with. And that will likely make you a better Linux admin, even if you decide to stick with it. Or maybe you’ll want to use the best tool for the job which could sometimes be Linux and sometimes a BSD. Getting to know a somewhat similar but also at times quite different *nix system will enable you to make an informed choice.

Not happy with Linux anymore in the recent years? Try out a BSD. If you need help to decide which one might be for you, I’ve written an article about that topic, too. Do a bit of reading, then install that BSD in a VM and explore. If you go with FreeBSD, make sure you take a look at the handbook (probably also available in your language) as that is a great source of information and one thing that sets FreeBSD apart from almost all Linux distros.

If you find that you like what you found, make a list of your requirements and find out if your BSD would indeed fulfill your needs. If it doesn’t, consider alternatives. Once the path is clear, I recommend to take a look at the community, too. For example there’s the weekly BSDNow! podcast that’s very informative. A lot of people have already written in, confessing that they are still Linux users only, but the topics of the show got them still hooked.

Do not rush things. Did you start with Linux or have you migrated e.g. from Windows? If you did come from a different OS, remember that there have been frustrating moments when you were all new to Linux and had certain misconceptions. You will be going through that again, but looking at the final outcome it will likely be a pretty rewarding journey.

Also don’t be shy and ask others if you don’t have the time or will to figure out everything yourself. The BSD people are usually pretty approachable and helpful. Feel free to ask me questions here, I might be able to give some answers.

It has been a couple of years now since I replaced the last machine that ran Linux at home. Would I choose to make the switch after all the experience that I gained since then? Oh yes! Anytime.

A glimpse into 2020

When you read this, the old year will be over (well, depending on the time zone you live in). If we’re lucky, this might be the year to get our hands on the first affordable RISC-V hardware that can actually run a Unix-like operating system. It should definitely be the year to get interesting devices like the ARM64-based PinePhone. And it also means that Python 2 is finally dead.

Speaking about that: For me 2019 has been a pretty busy year. On this blog I wrote about quite some different topics, among them my first attempt at writing something programming-related as I tried to teach myself a little bit of Python. If I had to name an overall theme, I’d say that the past year was the year of hardware architectures. I didn’t plan this, but that’s what’s happened. But I don’t actually want to look back in this post. On the contrary! But speaking of dead things also kind of fits into the next topic (more than I like…)!

FreeBSD on SPARC64

The next post that I plan to write will be about FreeBSD on the SPARC64 architecture. What I did not know when I decided on that is that it is more or less a doomed architecture when it comes to FreeBSD. SPARC64 is in grave danger – people expect support for it to be dropped before FreeBSD 13.0 is released!

The reason is that it is one of the architectures that still need the old GCC 4.2 (yes, from 2007!) toolchain – and that old cruft finally has to go. And while everybody agrees that this is a completely sensible thing to do, SPARC64 doesn’t seem to have as many friends among the FreeBSD developers to make the transition to something newer. A few people are trying to get something done (I’m also tinkering and trying to help), but it’s far from a save bet that it’ll succeed.

IMHO it would be a real shame to see FreeBSD on SPARC64 die. If it does survive, I’ll definitely try to help with QA. Can you help? If so: Please do! I’ll give more details on what the current status is and where the problems are in my January post.

FreeBSD on ARM64

I plan on writing another post about the current status on FreeBSD on ARM64. The topic of making it a Tier 1 architecture has recently been brought up again and I’d like to join the discussion about that rather sooner than later. If it wasn’t for the very unpleasant situation with SPARC64, this would actually be my next post.

Progress has been made on the networking issues on the Cavium ThunderX servers and I’ll also take a look again at the PineBook. Most likely I’ll also buy a PinePhone and/or one of their Tablets. If I do, you will find a review here.

Orchestration and Configuration Management with SaltStack

I wanted to write about this topic for a while, but now I’ve at last started to set aside the hardware that I need for the project. Several years ago (gosh…) I wrote a little comparison of Puppet, Chef, Salt and Ansible. After a general introduction I might update and publish this as well.

But that will only be the start of a series of posts introducing Salt. There will be a slight focus on FreeBSD, but in general it will show off how to work with various operating systems and distributions. We’ll start with Salt-SSH using Remote Execution Modules, talk about targeting and get to know grains. Then we’ll progress to the state system, pillar data and so on before switching to the master-minion model.

I’m looking forward to this one. If anybody has any ideas – just tell me, I’m open to suggestions on what to cover.

HardenedBSD

While I ran this derivative of FreeBSD for a couple of weeks on spare hardware (until I needed that for something else), I just didn’t find the time to write about my experience with it, yet. I liked it, though, and plan on re-visiting the OS. And when I do, you’ll read about it here for sure.

The project is currently being re-structured. So I’ll wait with this topic for a while longer. Might happen in the second half of the year (time flies by much too fast, anyway).

E-Mail

For years now I’ve been putting this one off. In fact I’ve started digging a bit into the topic twice. Two times I got distracted with other topics. Maybe third time is the charm?

To help making this more likely, I registered a domain (with a pretty bad pun in German) in December. Maybe now that I pay money, I’ll actually live up to the plan to “do moar with mail”. I’ll be using BSD technology where possible. So expect that the mailing stuff will involve OpenSMTPd.

illumos

This one is really a time issue. I’m still very much interested in the heritage of OpenSolaris and would like to do some more things with it. However I have no idea when I’ll find the time to do a dedicated illumos project. But there will definitely be some illumos involved with one of the other topics. You guess which one! 😉

Ravenports

The Ravenports project is still as fascinating to me as it was when I discovered it. I really wish I could dedicate more time to porting and helping to bring things forward (there’s still quite a lot of ports missing that I’d like to have and use).

While things are going well in general and ports are being updated really, really fast most of the time, big changes are rare right now. But big changes are what it makes sense to write about. And while there are some noteworthy things that I can think of, I’m still waiting for something else to land. Once that happens I will dedicate another post to Raven.

Linux vs. FreeBSD

I recently needed to setup a new Linux machine for a customer. Usually my co-workers do that, since I volunteered to take care of our BSD machines. That installation left me totally puzzled. Has the Debian installer become worse – or does my memory fail me and it has always been so bad (and I didn’t notice when I was into Linux only)?

Since then I thought about a few things. The conclusion is that I really love FreeBSD. It’s not perfect (well, nothing is), but there are so many areas where it’s much, much more comfortable to work with (can you say iptables or mdadm? Yuck!). And there is a lot more beauty and even technical genius if you take a closer look and compare things.

Yes, Linux in much more advanced in many areas. But that’s not much of a surprise given how much more manpower goes into that system. But it is a little miracle how the BSDs with their much lower manpower continue to deliver excellent operating systems on par with or even superior to Linux when it comes to sanity of use. Thank you, *BSD!

Happy new year!

So that’s what I have on my mind right now (I’m not out of ideas, but these are the topics that are on the top of my “would like to write about” list currently). Which of these topics will I be able to deliver and which will I miss? Time will tell. Feel free to comment and tell me what interests you the most.

Happy new year to all of my readers!