Ravenports: Status update and the Dragonfly case

This is part three of a series of posts on cross-platform package management. The previous posts contained general thoughts about software packaging today and a somewhat in-depth overview on the Ravenports package system.

In this post I want to give some more background on why Ravenports might be interesting to some people and explain the Dragonfly case. I’ll also give you a little status update.

Real-world use case: Package homogenization

As mentioned in the previous post, Ravenports currently supports DragonflyBSD, FreeBSD-amd64, Linux-glibc-x86_64 and Solaris/Illumos-amd64.

There are of course minor differences between the platforms: The shadow package is only available on Linux because the other systems use different ways for scrambling user passwords, there’s no fuse for Dragonfly because that is not supported there and iocage is FreeBSD-only because it’s a jail manager. Also some packages don’t build on the SunOS platform, yet, because they need additional patches. When I wrote my previous post, e.g. the xorg-server package was not available for SunOS. It is now, though it’s not of too much use, because the Xorg drivers aren’t. Still you can see things are moving in the right direction.

Other than those special cases, Raven is consistent across all supported platforms (which is one of the major features that caught my attention after all): You can get packages of the same version on Linux and FreeBSD (and the other platforms) and – as much as feasible – they are also configured alike. It’s not guaranteed that the official repositories hold the same package versions at all times. If you require that, you currently have to roll your own repos but that’s not hard to do at all (I’ll write about actually using Raven in detail in a future post).

As I’m working in a heterogeneous environment, it is my hope that Raven will take a lot of the headaches away that result from native package management in the Linux jungle (Ubuntu has one version, Debian ships another, CentOS is stuck with an ancient one and patches is to use a different directory structure, … you get the point) and as a bonus offers the same version on *BSD. I’m pretty sure that I’m not alone in this jungle and while most of us are able to survive there, few would deny that it does require a big machete and some teeth grinding from time to time. It’s ok – but can’t we use our time for better things? I think so.

The Dragonfly case

At home I’m more or less exclusively a FreeBSD user and that’s the platform that I do my porting work on. I have in fact never seriously run Dragonfly – not actually because of a lack of interest in it but rather due to a very limited set of hardware available to me. I’ve never found a spare piece of hardware among my pool that would run the OS well. Still I’m dedicating most of this post to DragonflyBSD. Why? Because Raven is in a special position there.

As you may know, DragonflyBSD is by far the smallest of the four “big” BSD projects (some actually talk about only three and leave out Dfly). They are doing an amazing job for the little manpower that they have, but all small projects struggle to keep up with changes going on in the open-source ecosystem. Run your desktop on a *nix machine? Simply ask your package manager how many packages you have installed to make that possible and get a rough idea on just how much work needs to be put into maintaining all of them.

Most OSes and distributions maintain their own source repositories. Dragonfly never had the manpower to even consider this (also they have way different focus than just doing again what everybody else does). Historically this is why they used Pkgsrc as this portable package system looked like a great option for Dragonfly. A lot of work was put into it, but there were quite some issues with it.

Pkgsrc on Dragonfly

Some were of technical nature: Pkgsrc can do binary packages, but if you know other package managers, the old tools really pale in comparison. There were conflicts in the release model, as Pkgsrc’s quarterly releases were not well fit for Dfly. But most importantly: A lot of packages were really outdated and where updates did occur so did breakage.

Updates were only tested against NetBSD and so it happened quite often that a single update broke anything from a couple to thousands (!) of ports for DragonflyBSD. Not even in the latter case would Pkgsrc suggest to revert the update that caused so much trouble – Dragonfly was expected to fix all of the fallout themselves. To be fair, there surely was no intention to break anything. But there simply weren’t any test farms either, so even if porters would have liked to care better for Dfly, it wouldn’t have been easy for them.

For technical problems there’s usually a solution, especially if people are involved who both are knowledgeable on the topic and show great dedication. What’s difficult to solve however, are political problems. And that’s what arose in the relationship between Dragonfly and Pkgsrc: DragonflyBSD has officially been a first-class citizen and thus on-par with NetBSD. But because of the frequent breakage, Dragonfly users felt differently about it. Unfortunately, multiple attempts and suggestions made to improve the situation also led to nothing.

In the end it became clear that Pkgsrc primarily is NetBSD’s package system. Since one of NetBSD’s primary goals is portability, Pkgsrc is of course also portable. This portability comes at a price, though – and that price is the need for a whole army of Pkgsrc maintainers to support an OS besides NetBSD well. As Pkgsrc was chosen due to the lack of manpower, this fact learned the hard way, showed that it wasn’t the right solution.

Dports

Searching for a different one, John Marino stepped forward in creating an alternative: Dports.

In a nutshell this meant bringing FreeBSD’s new package manager (pkg a.k.a. “pkg-ng”) to Dragonfly as well as FreeBSD’s ports – with the changes necessary to make those build on this platform. It was a huge task but the advantages over the old system were big enough for the project to make sense. Eventually DragonflyBSD ditched Pkgsrc after more than half a decade and adopted Dports. If you want to know more, you can e.g. read the old comments here.

Of course Dports is not a “do once” effort; it constantly requires work to keep all the ports in sync with newer versions in FreeBSD’s ports collection where the active development takes place. And John didn’t have enough, he continued to experiment with packaging, writing Synth. He didn’t like some of the aspects of FreeBSD’s ports collection and on the other hand wanted features that were unlikely to find their way into the FPC.

Then FreeBSD introduced flavored ports. According to John, Dports had in general been less work for more packages available in comparison to Pkgsrc. However breakage still is an issue. And since the flavor-related changes in FreeBSD this has become much worse. So over time things have become more pressing for a real and permanent solution.

Ravenports to completely replace Dports?

John had been thinking about new ways of package creation for a long time. With Synth he now had his own package building system and with package numbers increasing far beyond the point of somewhat acceptable maintainance requirements, he decided to give a new project a try after all and this evolved into what is Ravenports today.

Currently John Marino is working on Ravenports while still keeping Dports up to date. Last month he announced that he’d like to step down from working on Dports because he considers his time better spend on Raven. The big question is now: Will somebody volunteer to claim maintainership for Dports? So far it doesn’t look like it. Which means: After more than 5 years in existence, Dports might actually go away. Ravenports might replace it as the official packaging system on DragonflyBSD.

For that reason John has asked the community what packages are important for people. Again, I’m not a Dragonfly user, but I’ve been very surprised by the response of the community – or rather the lack of it. Only a few people have responded so far and this makes me wonder if the majority of Dragonfly users have either missed this totally or didn’t realize what it means.

I’m convinced that Ravenports definitely is the superior system. However a transition would have a huge impact. When Dragonfly switched from Pkgsrc to Dports it meant that a whole lot more package were available. In case of a possible switch to Ravenports the opposite would happen: There’s roughly 30,500 packages in Dports currently while at the moment the Dragonfly repository of Ravenports holds exactly 3,600 packages!

Yes, the numbers cannot be compared 1:1, but it’s not too hard to see that it would mean a dramatic decrease in packages being available. On the plus side, software versions in Ravenports are often much newer than the same program available on FreeBSD. More and more packages will be added to Raven – but this takes time. For that reason it’s very important that you tell us what software you depend on so that it can be added with higher priority.

What’s being done

Some packages have already been identified to be a problem. An important one is Firefox. Ravenports has it available in the latest ESR version that doesn’t require Rust. But web browsers don’t age very well and newer versions of Firefox are practically a must-have. Rust cannot be easily added, however – it’s another problematic package. A third example is everything TeX which is a beast of a project.

One user requested more components of the Xfce desktop. I had started porting Xfce for my previous article anyway and I’m planning to either port the complete desktop or at least the most important parts. Next is Thunar, Xfce’s file manager. But to give you an idea of what this means: Before I can even think about porting it, I need to get in a whole lot of dependencies first. A quick look at it showed that I might have to create up to 30 ports for that (including some that probably won’t be trivial). So that’s not work for one or two weeks but instead will likely take a lot longer.

Also I’m doing some work on getting FreeBSD-i386 working. It’s in fact mostly done; I have some changed ports that I need to commit and one last package doesn’t work to publish an i386 repo that is self-hosting. However this is more or less a thing that I’m doing for fun and to learn.

If I succeed with that, I plan on backporting Raven to FreeBSD 10 (amd64) and then 9. The reason for that is that I hope to make it run on MidnightBSD. It is a fork of FreeBSD and technically close to 9. As another (even smaller) BSD it also has the known problems in keeping up with all current package versions in their mports. So it could make sense to join efforts. But this is just an idea, I haven’t approached Lucas Holt, yet, and I won’t before I have something to show off.

Another platform that will likely be supported, is MacOS, further backing Ravenport’s promise of being a universal package system. And Linux support will be improved in the future as well. Currently distributions that ship very old versions of glibc (e.g. RedHat) cannot use Raven’s packages. Possible solutions to that are being discussed.

How YOU can help

Are you running a *nix operating system supported by Ravenports? In that case you can help. Do the bootstrap and start using it – it’ll install to /raven by default so there should be no conflicts with your other package system. If you’re using it, please provide feedback and create issues if you find problems! The more people actually use the packages, the more confident we can all be that those work well on all platforms.

If you have a little more interest in packaging systems, you can try to create a port and see if you like it. It does help if you’re familiar with FreeBSD’s ports, Dports or really any other package system, but that’s not a requirement. I started with practically no prior knowledge and now after less than a year, I’m maintaining over 80 ports. I’m not a programmer and don’t know much about Make and such. For me it has been a great learning experience.

Oh, and if I can do it, so can you. I intend to write another post on using Raven and at least one more about writing your own ports. If you want to give it a try before, feel free to contact me, I’ll try to help. Also any questions are appreciated so I get a better idea if what I should write about.

Advertisements

Ravenports: A modern, cross-platform package solution

This post is about Ravenports, a universal package system und building framework for *nix systems (DragonflyBSD, FreeBSD, Linux and Solaris at the time of this writing). It’s a relatively young project that begun in late February 2017 after a longer period of careful planning. The idea is to provide a unified, convenient experience in a cross-platform way while putting focus on performance, scalability and modern tooling.

What exactly is it and why should you care? If you’ve read my previous post, you know that I consider the old package systems lacking in several ways. For me Raven already does a great job at solving some problems existing with other systems – and it’s still far from tapping its full potential.

Rationale

A lot of people will think now: “We already have quite capable package systems. What’s the point in doing it again?” Yes, in many regards it’s “re-inventing the wheel”… And rightfully so! Most of the known package systems are pretty old now and while new features were of course added, this is sometimes problematic. There is a point where it’s an advantage to start fresh and incorporate modern ideas right from the start. Being able to benefit from the experience and knowledge gained by using the other systems for two decades when designing a new system is invaluable.

Ravenadm running on FreeBSD, OmniOS, Ubuntu Linux and DragonflyBSD

Ravenports was designed, implemented and is primarily maintained by a veteran in packaging software. John Marino at a time maintained literally thousands of ports for FreeBSD and DragonflyBSD. In addition to that, he wrote an alternative build tool called Synth. Aiming for higher portability, he modified Synth to work with Pkgsrc (which is available for many platforms) and also ported the modern Pkg package manager from FreeBSD to work with it.

In the end he had too many ideas about what could be improved in package building that would not fit into any existing project. Eventually Ravenports was born when he decided to give it a try and create a new framework with the powerful capabilities that he wanted to have and without the known weaknesses of the existing ones.

How does it compare to xyz?

It probably makes sense to get to know Ravenports by comparison to others. Let’s take a look at some of them first:

1) FreeBSD’s ports system is the oldest one such framework. It’s quite easy to use today, very flexible and since the introduction of Pkg (or “pkg-ng”) it also has a really nice package manager.
2) NetBSD adopted the ports system and developed it according to their own needs. It’s missing some of the newer features that FreeBSD added later but has gained support for an incredible amount of operating systems. Unfortunately it still uses the old pkg_* tools that really show their age now.
3) OpenBSD also adopted the early FreeBSD ports system. They took a different path and added other features. OpenBSD put the focus on avoiding users having to compile their own packages. To do so, they added so-called package flavors. This allows for building packages multiple times with different compile-time options set. Their package tools were re-written in Perl and do what they are meant to. But IMO they don’t compare well to a modern package manager.
4) Gentoo Linux with its portage system has taken flexibility to the extreme. It gives you fine-grained control over exactly how to build your software and really shines in that. The logical consequence is that, while it supports binary packages, this support is rudimentary in comparison.

EDE desktop, pekwm with Menda theme and brand-new LibreOffice

FreeBSD gained support for flavors in December 2017 and NetBSD did some work to support subpackages in a GSoC project in the same year. It’s hard to retrofit major new features into an existing framework, tough. When Ravenports started in the beginning of 2017, it already had those two features: Variant packages (Raven’s name for flavors) and subpackages. As a result they feel completely natural and fit well into the whole framework (which is why they are used excessively).

Ravenports knows ports options that can be set before building a package. Like with NetBSD or OpenBSD there’s generally fewer options available compared to FreeBSD. This is because Raven is more geared towards building binary packages than being a ports framework to build on the target machine (which would defeat the goal of always providing a clean building environment). For that reason the options mostly exist to support the variants for the packages. Compared to NetBSD’s Pkgsrc, Ravenports supports much fewer operating systems right now but has a much easier bootstrap process (binary!) for all supported platforms. It also offers a much superior package manager. When comparing against FreeBSD, OpenBSD and Gentoo, Ravenports is much more portable and supports multiple operating systems and – with the exception of FreeBSD – comes with a more modern package manager for binary packages.

Strong points

As Ravenports is not tied to a single operating system, it didn’t have to take into account specific needs that are for one OS only. In general there are no second-class citizens among the supported platforms. Also it was made to be agnostic of the package manager used. Right now it’s using Pkg only but other formats could be supported and thus binary packages be installed via pacman, rpm, dpkg, you-name-it.

Repology: Raven’s package freshness in percent (06/25/2018)

It allows for different versions of some software to be concurrently installed. If you e.g. want PHP 7.2 while some of your projects are stuck with 5.6 this is not a problem. It’s also possible to define a default version for databases like MySQL and Postgres as well as languages like Perl, Python and Ruby. Speaking of MySQL: Raven knows about Oracle MySQL, MariaDB, Percona and Galera. Only the first one is currently available (the ports for the others are missing) but the selection of which product to install is already present and the others can be easily added as needed.

If you build packages yourself you’ll notice that the whole tooling is fully integrated. Everything was planned right from the beginning to interact well and thus plays together just great. Also performance is something where Raven shines: Thanks to being programmed for high concurrency, operations like port scans are amazingly fast (if you know other frameworks).

Repology: Raven’s outdated package count (06/25/2018)

Raven follows a rolling-release model with extremely current package versions. In Repology, a fine tool for package maintainers and people interested in package statistics, Ravenports is the clear leader when it comes to freshness of the package repository: It rarely falls below 98% of freshness (while no other repo has managed to even reach 90% – and Repology lists almost 200 repositories!). If it does, it’s usually for less than a day until updates get pushed.

This is only possible because much of ports maintenance is properly automated. This saves a lot of work and allows for keeping the software version current without the need for dozens of maintainers. Custom port collections are supported if you have special needs like sticking to specific program versions. This way Raven can e.g. support legacy versions that should not be part of the main tree. It might also be interesting for companies that want to package their product for multiple platforms but need to keep the source closed. Ravenports supports private GitHub repositories for cases like this. All components of project itself are completely open-source, though, and are permissively licensed.

Also Raven is not the jealous kind of application. Packages are installed into /raven by default (you can choose to build your packages with a different prefix if you wish) and thus probably separate from the default system location for software. This makes it possible to use raven in addition to your operating system’s / distribution’s package manager instead of being forced to replace it.

Shortcomings

If you ask me about permanent problems with Raven: I don’t really see any. However there’s definitely a couple of things where it’s currently way behind other package systems. Considering how young the project is this is probably no wonder.

It’s a “needs more everything” situation. In fact it has the usual “chicken egg problem”: More available ports would be nice and potentially attract more users. With more users probably more people would become porters. And with more porters there’d surely be more ports available… But every new project faces problems like this and with resolve, dedication and perseverance as well as a fair amount of work, it’s possible to achieve the goal of making a project both useful and appealing enough for others to join in. Once that happens things get easier and easier.

KeePassXC, Geany and the EDE application menu

The Ravenports catalog has over 3,000 entries right now. It’s extremely hard to compare things like the package count, though. John provided an example: FreeBSD has 8 ports for each PostgreSQL version. With 5 supported versions that’s 40 ports. Ravenports has 5 ports with 8 subpackages each. In this case the package count is comparable, but not the port count. Taking flavors and multiversions into account, all repositories look much bigger than they actually are in case of available software. Also how to measure the quality of packages? What’s with ports that are used by less than a handful of people? What with those that are extremely outdated? Do you think they should count? It’s probably best to take a look and see if the software that you need is available. It is true though, that there’s of course still many important packages missing. IMO the most important one being Rust – which is not only needed for current versions of Firefox but increasingly important to build other software, too.

Also Linux support is not perfect, yet, and Solaris support even less so. On Solaris systems Raven is currently mostly binary-only because the Solaris kernel is unable to work with system libraries other than the ones matching exactly in version. Packages built on older releases of the OS work fine on newer ones, but for each OS release, a specific build environment would need to be created before building packages is possible. This is an issue that needs to be resolved in the future (I guess some help from the Illumos/Solaris community wouldn’t hurt). Also there are packages that don’t build on Solaris without patches which are not currently available. In case of important packages this leads to blockers since all other ports which depend on one such package also cannot be built: On FreeBSD there are 3,559 packages (including variants and metapackages) available from the repository at the present time. In the Solaris repo it’s only 2,851 packages. That’s certainly a nice start – but don’t expect to run a full-fledged desktop (or even X11 at all) there, yet!

In Linux land, distributions that come with glibc version 2.23 or newer work best. On distributions with older glibc versions (e.g. CentOS 7), software will not run as the standard C library is missing some required symbols. Raven will need to be bootstrapped again to support those distros. This is likely to happen before too long, but we’re not there, yet.

Current Firefox ESR version (+ sakura and pcmanfm in the panel)

MacOS (which might be supported soon), OpenBSD and NetBSD are not currently supported, nor is Linux with musl-libc or μclibc. Also currently Raven is amd64 only. ARM64 support is planned and i386 might or might not happen but are not available now.

Current status

At this time Raven is probably most interesting for people who love tech and enjoy tinkering on *nix systems as well as those who like the features and are ok with being early adopters. Yes, in general it’s ready for the latter. At least two people (including me) use Raven’s packages exclusively on one of their machines. I’d say it is ready as a daily driver (if you can live with the limited set of software available – or consider adding more ports). In fact I built a laptop that I use e.g. for on-call duty with it. Since that one is critical, it probably needs to be considered as “in production use”.

It’s possible to install various text mode applications with Raven, but X11 is also available. You can choose from multiple window managers or from at least two desktop environments (Lumina and the ultra-light EDE). Xfce4 is partially available (i.e. the panel is already ported). If you’re looking for web browsers, a current version of Firefox ESR (called “rustless-firefox”) can be installed as well as Surf, a simple webkit-based browser. The LibreOffice suite is available in its latest version, too. The same is true for the just released Perl 5.28 and Python 3.7.

Running Chocolate DooM and Chocolate Heretic

Oh, and if you’re into gaming… It’s not all just serious stuff. Yes, you can install and play DooM!

Conclusion

Ravenports is a fascinating project with lots and lots of possibilities. I wanted to get into porting with FreeBSD for quite a while but hesitated as I’m not a programmer. Then again I had been interested in package building for a long time and had played around with it on Arch Linux quite a bit. After my submissions to FreeBSD had been rotting in bug tracker for months (and still are after almost a year), I chose to give Raven a try in the meantime.

I was already familiar with Pkg and had used Synth before, too. Bootstrapping Raven’s pkg and then installing stuff was as easy as expected. The same was true for building the ports myself. Then I did quite a bit of reading and wrote my first port. It didn’t take more than 5 minutes after I opened my pull request on GitHub, before John responded – and the port was committed not much later. This was such a huge contrast that I decided to do more with Raven.

There was a learning curve, yes, but I received lots of help in getting started. I obviously liked the project enough to become a regular contributor and even got commit access to the ravensource repo later. Currently I’m maintaining just over 80 ports and I hope to write many more in the future. There have been some hard ports along the way (where I learned a lot about *nix), but lots of things are actually pretty easy once you get the hang of it.

Tongue-in-cheek: Make chaos or “make sense”!

If this post got you interested, just give it a try. Feel free to comment here and if you run into problems I’ll try to help. After this general overview of Raven the next post I plan to write will be on actually using it.