[New to Gemini? Have a look at my Gemini FAQ.]
This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2021/freebsd_package_building_pt4.gmi
In the previous posts of this series, there was an introduction to package building on FreeBSD and we discussed basic Synth usage. The program’s configuration, working with compiler cache and using the tool to update the installed applications was covered as well.
This article is about some advanced functionality of Synth like using web reports, building package sets and serving repositories as well as taking a look at logs.
Reporting
Synth comes with very useful reporting reporting functionality. For the default LiveSystem profile, Synth writes its logs to /var/log/synth. There it also creates a subdirectory called Reports and puts an advanced web report in there. It looks like this:
% ls -1 /var/log/synth/Report 01_history.json favicon.png index.html progress.css progress.js summary.json synth.png
We’re going to build and setup a web server to make it accessible. I will use OpenBSD HTTPd for a reason that we’re going to talk about in a minute (besides me liking it quite a bit). Let’s use Synth to install it first. Then we’re going to enable it and create a log directory for it to use:
# synth install www/obhttpd # sysrc obhttpd_enable=YES # mkdir /var/log/obhttpd
OpenBSD HTTPd successfully installed by Synth
Alright. Remember how I told you not to change Synth’s directory configuration unless you have a reason for it? Now we have: We’re going to serve both the web report and the packages over http and we’re using OpenBSD HTTPd for that. That browser is chrooted: Not just by default, BTW but by design! You cannot turn it off. Unless told otherwise, Synth has a directory structure that doesn’t fit this scenario well. So we’re going to change it.
First thing is creating a new directory for the logs and then changing the configuration so Synth uses that:
# mkdir -p /var/synth/www/log > # synth configure
choc
Change setting E to /var/synth/www/log and save. The new directory will of course be empty. We could copy the old files over, but let’s just rebuild HTTPd instead. Using synth build or synth just-build doesn’t work, though; the tool will detect that an up to date package has already been built and that nothing needs to be done. That’s what the force command is handy for:
# synth force www/obhttpd
Web server setup for reports
Now that we have something to serve, we can edit the webserver’s configuration file. Simple delete everything and put something like this into /usr/local/etc/obhttpd.conf:
chroot "/var/synth/www"
logdir "/var/log/obhttpd"
server synth.local {
listen on * port 80
root "/log"
log style combined
location "/" {
block return 302 "http://$HTTP_HOST/Report/index.html"
}
}
This defines where the chroot begins and thus where the unprivileged webserver processes are contained. It also defines the log directory as well as a virtual host (simply called “server” in OpenBSD HTTPd) with the name “synth.local”. Either replace this with a proper domain name that fits your scheme and configure your DNS accordingly or use /etc/hosts on all machines that need to access it to define the name there.
The server part is pretty simple, too. It makes HTTPd bind on port 80 on every usable network interface. The web root is defined relative to the chroot, so in this case it points to /var/synth/www/log. I’ve grown a habit of using the more detailed combined log style; if you’re fine with the default common format, you can leave the respective line out. Finally the configuration block defines a special rule for location / which means somebody accesses the virtual host directly (i.e. http://synth.local in this case). It will make the browser be redirected to the report index instead. Getting a special file (like e.g. lang___python37.log in the log directory) will not trigger the rule and thus still work. This is just a convenience thing and if you don’t like it leave it out.
All that’s missing now is starting up the webserver:
# service obhttpd start
You should now be able to point your browser at the the vhost’s name (if you made it resolve). Just using the machine’s IP address is also going to work in this case since it’s the default vhost. But better make it reachable using the configured name as we’re adding another vhost in a moment.
Synth web report for the latest package run
Authentication
But for now what about security? Let’s say you don’t want to share your report with the whole world. One easy means of protecting it is by using HTTP basic auth. OpenBSD HTTPd uses standard .htpasswd files. These can however use various cryptographic hashes for the passwords – whereas HTTPd only supports one: bcrypt.
The first time I tried to do authentication with OpenBSD HTTPd, it drove me completely nuts as I couldn’t get it working. Fortunately I own Michael W. Lucas’ excellent book “Httpd and Relayd Mastery”. After digging it out the index indicated that I might want to read page 28. I did, banged my head against the table and immediately got it working using that hashing algorithm. Don’t be like me, skip trying to use foreign tools you may be used to and just do it right in the first place. HTTPd comes with its own htpasswd binary. Use that.
In this example I’m adding a user called “synth”. Use whatever you prefer. Then give the password two times. This leaves you with a valid .htpasswd file that HTTPd could use – if it was allowed to access it! Let’s fix that problem:
# chown root:wheel /var/synth/www/.htpasswd > # chmod 0640 /var/synth/www/.htpasswd
Having the authentication information in place, we only need to add another location block to the webserver’s vhost configuration. Put the following in there after the line that closes the previous location block:
location "/Report/* {
authenticate with "/.htpasswd"
}
Note the htpasswd file’s location! It’s within the chroot (or it couldn’t be accessed by the webserver), but outside the webroot directory. So HTTPd could never accidentally serve it to somebody who knew that it was there and requested the file.
The only thing that remains is restarting the webserver. Next time you visit the report page, you’ll be asked to authenticate first.
# service obhttpd restart
Package repository
So far all of our packages have been created in a directory outside of the webserver’s chroot. If we want to make them available via HTTP, we need to use another path for them. Therefore we’re going to create a directory and reconfigure Synth again:
# mkdir -p /var/synth/www/packages # synth configure
This time it’s setting B. Change it to /var/synth/www/packages and save. Now let’s build a package that draws in a couple of dependencies:
# synth just-build chocolate-doom
We can watch it now via the web reports while it’s building. Since it’s a new directory where no packages exist, yet, Synth is first going to build the package manager again. During this early stage no report is available, but once that’s finished the reports work.
While we’re rebuilding all packages due to the new package directory, Synth can take advantage of ccache as we haven’t modified it’s path. Wonder how much of a difference that actually makes? Building llvm10 on its own, one time using the cache and one time (for testing purposes) without it will show you the difference:
Duration: 00:13:32 (with ccache)
Duration: 02:09:37 (uncached)
Synth web report while it’s building
It gives us all the information that the curses UI holds – and more. The number of entries for completed packages can be changed. You can browse those page-wise back to the very first packages. It’s possible to use filters to e.g. just list skipped or failed packages. You can view / download (whatever your browser does) the log files for all those packages. And there’s even a search (which can be very handy if you’re building a large package set).
Report with only 10 entries per page
As long as packages are being built, the report page also shows the builder information and automatically refreshes the page every couple of seconds. Once it completes, it removes builder info (which would only waste space) and stops the polling. You can always come back later and inspect everything about the latest package run. The next one will overwrite the previous information, though.
Synth report search function
Now that we have a bunch of newly built packages, let’s see what that looks like:
# ls -1 /var/synth/packages/All autoconf-2.69_3.txz autoconf-wrapper-20131203.txz automake-1.16.3.txz binutils-2.33.1_4,1.txz bison-3.7.5,1.txz ca_root_nss-3.63.txz ccache-3.7.1_1.txz celt-0.11.3_3.txz chocolate-doom-3.0.1.txz cmake-3.19.6.txz curl-7.76.0.txz db5-5.3.28_7.txz docbook-1.5.txz docbook-sgml-4.5_1.txz docbook-xml-5.0_3.txz docbook-xsl-1.79.1_1,1.txz doom-data-1.0_1.txz evdev-proto-5.8.txz expat-2.2.10.txz flac-1.3.3_1.txz [...]
Showing only ignored packages in the report (none in this case)
The packages are there. But what’s in the main directory?
# ls -l /var/synth/www/packages total 18 drwxr-xr-x 2 root wheel 150 Jun 7 23:57 All drwxr-xr-x 2 root wheel 3 Jun 7 23:21 Latest
This is not a valid pkg(8) repository. Which is no wonder since we used just-build. So we’re going to have Synth create an actual repository from these packages next:
Searching in the report after the build was completed
# synth rebuild-repository # ls -l /var/synth/www/packages total 117 drwxr-xr-x 2 root wheel 150 Jun 7 23:57 All drwxr-xr-x 2 root wheel 3 Jun 7 23:21 Latest -rw-r--r-- 1 root wheel 163 Jun 8 00:02 meta.conf -rw-r--r-- 1 root wheel 236 Jun 8 00:02 meta.txz -rw-r--r-- 1 root wheel 40824 Jun 8 00:02 packagesite.txz
Here we go, that’s all that pkg(8) needs. Synth should have automatically updated your repository configuration to use the new location. Have a look at /usr/local/etc/pkg/repos/00_synth.conf – the URL should point to the new directory.
Serving the repository
The next step is to make the repository available in the network, too. So edit /usr/local/etc/obhttpd.conf once more and add another “server” (i.e. vhost):
server pkg.local {
listen on * port 80
root "/packages"
log style combined
location * {
directory auto index
}
}
One service restart later you should be able to access the repository via a web browser from any machine in the same subnet (if you got your DNS right):
# service obhttpd restart
Looking at the package repository with a browser
This is already it, but let’s prove that it works, too. I’m adding the “pkg.local” name to the machine’s 127.0.0.1 definition in /etc/hosts, then change the URL in the Synth repo to fetch packages via HTTP:
url : http://pkg.local,
I’ve also created a FreeBSD.conf to disable the official repository. Let’s stop the webserver for a second and then try to update the package DB:
# service obhttpd stop # pkg update Updating Synth repository catalogue... pkg: Repository Synth has a wrong packagesite, need to re-create database pkg: http://pkg.local/meta.txz: Connection refused Unable to update repository Synth Error updating repositories!
Ok, so there’s no other repository configured anymore and this one is not accessed via the filesystem. So we’re going to start the webserver once more (give it a sec) and then try again:
# service obhttpd start # pkg update Updating Synth repository catalogue... Fetching meta.conf: 100% 163 B 0.2kB/s 00:01 Fetching packagesite.txz: 100% 40 KiB 40.8kB/s 00:01 Processing entries: 100% Synth repository update completed. 148 packages processed. All repositories are up to date.
Great! So now we can install DooM on this box or on any other machine running FreeBSD 13.0 which can reach it over the network.
Package sets
So far we’ve only either built all packages for what was already installed on the machine or for single ports that we selected at the command line. But now that we can serve packages over the network, it’s rather tempting to use a powerful build machine to build packages for various other FreeBSD machines, isn’t it? Let’s assume that you’re going to share packages with a KDE lover.
First we should prepare a list of packages that we need, starting with what is installed on our machine.
# pkg info | wc -l 345
Wow, that’s already quite some packages for such a pretty naked system! But we don’t need to consider them all as most of them are just dependencies. Let’s ask pkg(8) for the origin of all packages we explicitly installed (i.e. which were not recorded as automatically installed):
# pkg query -e %a=0 %o > /var/synth/pkglist # cat /var/synth/pkglist x11-wm/awesome devel/ccache graphics/drm-kmod devel/git www/obhttpd ports-mgmt/pkg ports-mgmt/portmaster x11/sakura x11/setxkbmap security/sudo ports-mgmt/synth sysutils/tmux x11/xfce4-screenshooter-plugin x11/xorg-minimal
That’s better! But we certainly don’t need portmaster anymore, so we can take it off the list (and deinstall it actually). Let’s add www/firefox and x11/kde5 for our pal (and sort the list since it’s a bit ugly right now).
Once that’s done, we should be able to do a simple:
# synth build /var/synth/pkglist Error: port origin 'devel/git' not recognized. Perhaps you intended one of the following flavors? - devel/git@default - devel/git@gui - devel/git@lite - devel/git@svn - devel/git@tiny
Oh yes, right! We need to edit our list and specify the flavor to build! I’m going with the lite variant here, so the git line needs to be changed to this:
devel/git@lite
Then we can try again – and yes, it starts building after calculating the required dependencies.
Logs
Whoopsie! After about an hour stupid me removed the network cable for a moment. This has caused a couple of build failures (see screenshot). The report will display the phase that the build failed. In this case it’s the fetch phase (and we don’t have to look for the reason as we already know it). Sometimes a distfile mirror is temporary down or the distfile has been removed. In that case you will have to manually get the files and put them into the distfiles directory. Skipped ports also display the reason, i.e. which dependency failed previously.
Failed and skipped ports due to a connection problem
I better re-attach that cable right away and start the building over… Many hours later it has finished. But what’s this? Rust has failed again (and this time it wasn’t me)! And it failed at the stage phase. When this happens it’s usually because of a broken port got committed. Update your ports tree and hope that it has been fixed in the meantime. This is not the reason in our case, however.
Another phase, another failure!
But how do we find out what actually happened? Well, by looking at the logs, of course. Here’s the last 15 lines of the relevant log:
finished in 183.141 seconds
< Docs { host: TargetSelection { triple: "x86_64-unknown-freebsd", file: None } }
Install docs stage2 (Some(TargetSelection { triple: "x86_64-unknown-freebsd", file: None }))
running: "sh" "/construction/xports/lang/rust/work/rustc-1.51.0-src/build/tmp/tarball/rust-docs/x86_64-unknown-freebsd/rust-docs-1.51.0-x86_64-unknown-freebsd/install.sh" "--prefix=/construction/xports/lang/rust/work/stage/usr/local" "--sysconfdir=/construction/xports/lang/rust/work/stage/usr/local/etc" "--datadir=/construction/xports/lang/rust/work/stage/usr/local/share" "--docdir=/construction/xports/lang/rust/work/stage/usr/local/share/doc/rust" "--bindir=/construction/xports/lang/rust/work/stage/usr/local/bin" "--libdir=/construction/xports/lang/rust/work/stage/usr/local/lib" "--mandir=/construction/xports/lang/rust/work/stage/usr/local/share/man" "--disable-ldconfig"
install: creating uninstall script at /construction/xports/lang/rust/work/stage/usr/local/lib/rustlib/uninstall.sh
install: installing component 'rust-docs'
### Watchdog killed runaway process! (no activity for 78 minutes) ###
--------------------------------------------------
-- Termination
--------------------------------------------------
Finished: Wednesday, 9 JUN 2021 at 00:04:47 UTC
Duration: 04:27:03
Ha! The build process was killed by the watchdog! Bad doggy? It does happen that the process would eventually have finished. Not this time. We have to dig a little deeper. In /var/log/messages of the build machine I can find the messages kernel: swap_pager: out of swap space and kernel: swp_pager_getswapspace(4): failed. This machine has 24 GB of RAM and 8 GB of swap space configured. And by building 6 huge ports concurrently, it exceeded these resources! Keep in mind that package building can be quite demanding, especially if you use tmpfs (which you should if you can).
So, there we are. We’ve configured our build server for web reports and serving the repository. We’ve looked at building package sets and covered a few examples of what can go wrong. And that’s it for today.
What’s next?
The last article about Synth will cover make.conf, signing repositories and using cron for automated builds. We’ll also take a brief look at profiles.
eerielinux 
One thought on “FreeBSD package building pt. 4: (Slightly) Advanced Synth”