One year of flying with the Raven: Ready for the Desktop?

It has been a little over one year now that I’m with the Ravenports project. Time to reflect my involvement, my expectations and hopes.

Ravenports

Ravenports is a universal packaging framework for *nix operating systems. For the user it provides easy access to binary packages of common software for multiple platforms. It has been the long-lasting champion on Repology’s top 10 repositories regarding package freshness (rarely dropping below 96 percent while all other projects keep below 90!).

For the porter it offers a well-designed and elegant means of writing cross-platform buildsheets that allow building the same version of the software with (completely or mostly) the same compile-time configuration on different operating systems or distributions.

And for the developer it means a real-world project that’s written in modern Ada (ravenadm) and C (pkg) – as well as some Perl for support scripts and make. Things feel very optimized and fast. Not being a programmer though, I cannot really say anything about the actual code and thus leave it to the interested reader’s judgement.

If you’re interested in a more comprehensive introduction to Ravenports, I’ve written one half a year ago.

Platforms

Ravenports has initially been developed on DragonFly BSD. When I became aware of it, it had already been ported to work on Linux, too. I liked the idea of the project, but had no DF or Linux boxes available for tinkering and didn’t feel like setting one up. Thus I moved on.

As I checked back a little later, FreeBSD support had been added. Since I had just lost my excuse not to try it out right away, I started playing with it – and was pretty happy. At that time I had trouble to get a port that I wrote into FreeBSD’s Ports Collection and thought that Raven could be an excellent playground to learn something and get a bit of experience that might help me later with FreeBSD.

The Xfce4 desktop – installed via Raven

I’ve long changed my mind, though! Raven is rather similar to FreeBSD’s ports system in many ways but where it differs it’s clearly superior. Also I love the cross-platform aspect and thus Raven is simply the better place for me to make home.

This year saw the introduction of Solaris/Illumos support that I tried out on OmniOS. Also Darwin support landed, upping the count of supported platforms to 5 already! Not too bad for a young project, huh? While Raven does work on all five platforms now it does so to varying degrees. But more on that later.

General activity

The Ravenports project consists of multiple Git repositories hosted on GitHub. The first one is Ravensource which most importantly holds the “raw” ports as they are written by the porters. It’s the most busy repo with over 5.200 commits since March 2017 (including almost 500 by me).

Then there’s the actual Ravenports repo that mostly contains the buildsheets which are compiled from Ravensource. It has over 1.400 commits right now.

Installing the xfce-single-core meta-package

Finally there’s the repo for the Ravenadm command-line tool. It’s approaching 900 commits since February 2017.

There’s still more to Raven like the Pkg package manager from FreeBSD (that was modified to add Zstd compression support) or libbsd4sol, a portability library which allows building code on Solaris that uses BSDisms (which was needed to add support for that platform to Raven). Most of the work on all repos was done by John alone.

With over 100 pull requests and more than 20 issues it’s clear now that there’s some interest in the project. Raven is still very small, though, with 6 people haveing contributed ports so far. After learning the basics and opening pull requests for half a year, I’ve been granted write-access to the source repository. Just recently I was able to push my 100th active port (there have been ports that became obsolete and were removed).

In general I’d say that there could of course be more people around and that the project would benefit from being able to provide more packages – though more than 3.200 is not bad at all! Also it’s good that there seems to be a growing user base which is even more important than having more porters join in. From my point of view, Raven is a healthy and fast-moving project. Still young, but doing well and heading in the right direction.

Major changes

There have been some pretty big changes that happened with Raven over time. Initially John started with a GCC6-based toolchain, only to switch to GCC7 when that was released. That was before my time with the project, but I witnessed the switch to GCC8.

Changing the toolchain certainly is a major interruption and most people are advised to just wait for the official repository to be re-rolled and then update. I had some bad luck in this regard – literally the day after I finally completed a working (and almost complete) set of basic packages for the FreeBSD_i386 platform, I faced the change to GCC8. Due to a lack of time I still haven’t repeated the switch on i386 (but I still plan to do it sometime).

The thunar file manager

Other changes that always have a huge impact (causing lots and lots of packages to be rebuilt) is adopting a new version (as well as dropping an old one) of the popular interpreter languages like Python, Perl and Ruby. Ravenports always supports two versions of Perl and Ruby and two versions of Python 3 (as well as 2.7 for now). So when Python 3.7 was released, 3.5 was removed and Perl 5.24 had to go when 5.28 was added.

Recently the former LLVM port that included everything regarding LLVM was split (LLVM, Clang, lld, openmp). Also now and then new statements are added to Ravenadm, so that old versions cannot work with a new release of the buildsheet repository (which is called “conspiracy”). But this is pretty easy to work around compared to the changes mentioned before.

So on the whole, Raven has proven that it can easily stand even big changes. For me this is essential to build faith in a project. And Raven is doing well in this regard.

Desktop-ready?

There are lots of people who will want to use Raven on servers. That’s totally fine of course. But for a project as ambitious as Ravenports, it’s necessary to provide a somewhat comfortable environment for the developers and the users alike. If it doesn’t manage to become a daily driver for people it cannot succeed.

For that reason I decided to work towards good desktop support for the little dev machine that I dedicated to my work on the project. When I started, X11 was already working and Openbox had freshly landed in the repos. So I had a simplistic environment to work with: Openbox + Xterm. However I could not even change my keyboard layout! Therefor I wrote a port for setxkbmap and eventually it was accepted as the first outside contribution to the project.

The Surf web browser

Next I did some work to get the FLTK toolkit and the EDE desktop in. Then I added my favorite terminal emulator, Sakura. This worked out pretty well and the biggest shortcoming at the end of 2017 was that there was no real graphical browser available. A lot has changed since then!

Desktop choices

Today you can choose between multiple window managers, both floating and tiling:

  • twm
  • cwm
  • openbox
  • fluxbox
  • xfwm4
  • pekwm
  • i3

And in case you prefer a real desktop environment, there are also several available:

  • Lumina (moderate, Qt-based)
  • Xfce4 (somewhat light-weight, GTK-based)
  • EDE (extremely frugal and minimalistic, FLTK-based)

Two graphical web browsers are available, Surf (which is deliberately simplistic and does not even support tabs) as well as an old version of Firefox (the last one that builds without Rust). This is certainly not perfect but much better than a year before.

Also other important programs are available, including LibreOffice! Last month the Apache webserver landed – which is a pretty complex port compared to many others.

Shortcomings

Are there packages you’ll miss? Most certainly. However there’s a wishlist now with ports that people would like to see created (please feel free to add more requests there). And that’s another good step ahead. Currently it’s almost 120 items long. Fortunately there’s been some success, too, and 26 requested ports have been created and taken of the list so far.

There are some future ports that will require lots of effort (hint: Help wanted!). The most important one that blocks some other important ports is the Rust compiler. There has been some work done on this but it’s not done, yet. Another real beast is TeX. This totally must be supported at some point. Current versions of Firefox and Chromium are often asked for. And somebody even requested Eclipse (which needs Java!). So there’s definitely more than enough work to do.

Using Raven on Linux works, but there are some flaws. Initially the Pkg package manager used to crash quite often. John traced that back to a bug in the version of SQlite that’s used internally by Pkg: The problem only struck on Linux and was fixed by using a newer version instead. While it’s much better now, there’s still the occasional problem with it.

While the packages from the repo work finde on Solaris 10u8 and above as well als Illumos, the exact version 10u8 is currently required to build packages. This is due to Solaris not being able to work with older system libraries in the build chroot. It would be great to haven an alternative ravensys-root for any Illumos distribution (OmniOS, SmartOS, Tribblix, …) available so that interested people without access to that specific closed-source Solaris version can develop Raven on that platform.

I don’t know how well Raven works on Darwin. Since I don’t have access to any macOS machines and PureDarwin is not really ready, yet, there’s currently no chance for me to test it. I intend to buy an older MacBook or something in the future, though, if I come across a fair offer and have some money available to spend on my hobby.

Some ports are not available on one platform or the other: Illumos mostly because they’d require patches to build and Linux often because it relies on additional libraries that have not yet been added to Raven. And then there’s a lot of packages that are mostly untested. All of these issues can be fixed, of course. All of those require a larger user-base, though. So it’s probably the best strategy to keep working on making Raven attractive to more users and address things when the right people show up.

What’s to come?

Currently Raven uses the primordial X11 input drivers (xf86-input-keyboard and xf86-input-mouse) on all platforms. In 2013 Linux pioneered support for generic input drivers by exposing the kernels “event devices”. Not too much later many Linux distributions adopted xf86-input-evdev. In 2014 there was a GSOC project to add evdev support for FreeBSD. Like many projects it came along a good part of the way but eventually was left unfinished. It was picked up and completed by a FreeBSD developer in 2016.

Xfce’s settings and applications menu

To use it, a special kernel had to be built so it would expose /dev/input device nodes. Then a sysctl had to be set – and eventually X11 had to be patched for emulated udev support… Why would anybody want to do all this just for different input drivers? Multi-touch support is just one valid reason. Another one is that having evdev-based input drivers is half the way to eventually support libinput, too. And that is one of the prerequisites for Wayland!

This month FreeBSD has finally enabled evdev support in the GENERIC kernel in both -CURRENT and 12-STABLE. That means the upcoming FreeBSD 12.0 will not support it out of the box, but most likely a future 12.1 will. Dragonfly BSD has also grown support for event devices and people are interested in working towards Wayland. I hope that we’ll be able to get xf86-input-evdev working with our X11 (on Dragonfly, FreeBSD and Linux) next year,

I’m taking a little break from Xfce now (but plan to port most of the remaining components later to make it a well-supported DE in Raven). There are a few things I have planned like adding Linux support for OpenVPN (it depends on some libraries and programs that are Linux only which are not yet in Raven). Also I intend to take a look at adding some more Qt5 components and write a few requested ports. And finally I want to write another post next year – a tutorial on using Ravenports and creating new ports.

So keep flying with us – it’s exciting times!

Advertisements

Ravenports: A modern, cross-platform package solution

This post is about Ravenports, a universal package system und building framework for *nix systems (DragonflyBSD, FreeBSD, Linux and Solaris at the time of this writing). It’s a relatively young project that begun in late February 2017 after a longer period of careful planning. The idea is to provide a unified, convenient experience in a cross-platform way while putting focus on performance, scalability and modern tooling.

What exactly is it and why should you care? If you’ve read my previous post, you know that I consider the old package systems lacking in several ways. For me Raven already does a great job at solving some problems existing with other systems – and it’s still far from tapping its full potential.

Rationale

A lot of people will think now: “We already have quite capable package systems. What’s the point in doing it again?” Yes, in many regards it’s “re-inventing the wheel”… And rightfully so! Most of the known package systems are pretty old now and while new features were of course added, this is sometimes problematic. There is a point where it’s an advantage to start fresh and incorporate modern ideas right from the start. Being able to benefit from the experience and knowledge gained by using the other systems for two decades when designing a new system is invaluable.

Ravenadm running on FreeBSD, OmniOS, Ubuntu Linux and DragonflyBSD

Ravenports was designed, implemented and is primarily maintained by a veteran in packaging software. John Marino at a time maintained literally thousands of ports for FreeBSD and DragonflyBSD. In addition to that, he wrote an alternative build tool called Synth. Aiming for higher portability, he modified Synth to work with Pkgsrc (which is available for many platforms) and also ported the modern Pkg package manager from FreeBSD to work with it.

In the end he had too many ideas about what could be improved in package building that would not fit into any existing project. Eventually Ravenports was born when he decided to give it a try and create a new framework with the powerful capabilities that he wanted to have and without the known weaknesses of the existing ones.

How does it compare to xyz?

It probably makes sense to get to know Ravenports by comparison to others. Let’s take a look at some of them first:

1) FreeBSD’s ports system is the oldest one such framework. It’s quite easy to use today, very flexible and since the introduction of Pkg (or “pkg-ng”) it also has a really nice package manager.
2) NetBSD adopted the ports system and developed it according to their own needs. It’s missing some of the newer features that FreeBSD added later but has gained support for an incredible amount of operating systems. Unfortunately it still uses the old pkg_* tools that really show their age now.
3) OpenBSD also adopted the early FreeBSD ports system. They took a different path and added other features. OpenBSD put the focus on avoiding users having to compile their own packages. To do so, they added so-called package flavors. This allows for building packages multiple times with different compile-time options set. Their package tools were re-written in Perl and do what they are meant to. But IMO they don’t compare well to a modern package manager.
4) Gentoo Linux with its portage system has taken flexibility to the extreme. It gives you fine-grained control over exactly how to build your software and really shines in that. The logical consequence is that, while it supports binary packages, this support is rudimentary in comparison.

EDE desktop, pekwm with Menda theme and brand-new LibreOffice

FreeBSD gained support for flavors in December 2017 and NetBSD did some work to support subpackages in a GSoC project in the same year. It’s hard to retrofit major new features into an existing framework, tough. When Ravenports started in the beginning of 2017, it already had those two features: Variant packages (Raven’s name for flavors) and subpackages. As a result they feel completely natural and fit well into the whole framework (which is why they are used excessively).

Ravenports knows ports options that can be set before building a package. Like with NetBSD or OpenBSD there’s generally fewer options available compared to FreeBSD. This is because Raven is more geared towards building binary packages than being a ports framework to build on the target machine (which would defeat the goal of always providing a clean building environment). For that reason the options mostly exist to support the variants for the packages. Compared to NetBSD’s Pkgsrc, Ravenports supports much fewer operating systems right now but has a much easier bootstrap process (binary!) for all supported platforms. It also offers a much superior package manager. When comparing against FreeBSD, OpenBSD and Gentoo, Ravenports is much more portable and supports multiple operating systems and – with the exception of FreeBSD – comes with a more modern package manager for binary packages.

Strong points

As Ravenports is not tied to a single operating system, it didn’t have to take into account specific needs that are for one OS only. In general there are no second-class citizens among the supported platforms. Also it was made to be agnostic of the package manager used. Right now it’s using Pkg only but other formats could be supported and thus binary packages be installed via pacman, rpm, dpkg, you-name-it.

Repology: Raven’s package freshness in percent (06/25/2018)

It allows for different versions of some software to be concurrently installed. If you e.g. want PHP 7.2 while some of your projects are stuck with 5.6 this is not a problem. It’s also possible to define a default version for databases like MySQL and Postgres as well as languages like Perl, Python and Ruby. Speaking of MySQL: Raven knows about Oracle MySQL, MariaDB, Percona and Galera. Only the first one is currently available (the ports for the others are missing) but the selection of which product to install is already present and the others can be easily added as needed.

If you build packages yourself you’ll notice that the whole tooling is fully integrated. Everything was planned right from the beginning to interact well and thus plays together just great. Also performance is something where Raven shines: Thanks to being programmed for high concurrency, operations like port scans are amazingly fast (if you know other frameworks).

Repology: Raven’s outdated package count (06/25/2018)

Raven follows a rolling-release model with extremely current package versions. In Repology, a fine tool for package maintainers and people interested in package statistics, Ravenports is the clear leader when it comes to freshness of the package repository: It rarely falls below 98% of freshness (while no other repo has managed to even reach 90% – and Repology lists almost 200 repositories!). If it does, it’s usually for less than a day until updates get pushed.

This is only possible because much of ports maintenance is properly automated. This saves a lot of work and allows for keeping the software version current without the need for dozens of maintainers. Custom port collections are supported if you have special needs like sticking to specific program versions. This way Raven can e.g. support legacy versions that should not be part of the main tree. It might also be interesting for companies that want to package their product for multiple platforms but need to keep the source closed. Ravenports supports private GitHub repositories for cases like this. All components of project itself are completely open-source, though, and are permissively licensed.

Also Raven is not the jealous kind of application. Packages are installed into /raven by default (you can choose to build your packages with a different prefix if you wish) and thus probably separate from the default system location for software. This makes it possible to use raven in addition to your operating system’s / distribution’s package manager instead of being forced to replace it.

Shortcomings

If you ask me about permanent problems with Raven: I don’t really see any. However there’s definitely a couple of things where it’s currently way behind other package systems. Considering how young the project is this is probably no wonder.

It’s a “needs more everything” situation. In fact it has the usual “chicken egg problem”: More available ports would be nice and potentially attract more users. With more users probably more people would become porters. And with more porters there’d surely be more ports available… But every new project faces problems like this and with resolve, dedication and perseverance as well as a fair amount of work, it’s possible to achieve the goal of making a project both useful and appealing enough for others to join in. Once that happens things get easier and easier.

KeePassXC, Geany and the EDE application menu

The Ravenports catalog has over 3,000 entries right now. It’s extremely hard to compare things like the package count, though. John provided an example: FreeBSD has 8 ports for each PostgreSQL version. With 5 supported versions that’s 40 ports. Ravenports has 5 ports with 8 subpackages each. In this case the package count is comparable, but not the port count. Taking flavors and multiversions into account, all repositories look much bigger than they actually are in case of available software. Also how to measure the quality of packages? What’s with ports that are used by less than a handful of people? What with those that are extremely outdated? Do you think they should count? It’s probably best to take a look and see if the software that you need is available. It is true though, that there’s of course still many important packages missing. IMO the most important one being Rust – which is not only needed for current versions of Firefox but increasingly important to build other software, too.

Also Linux support is not perfect, yet, and Solaris support even less so. On Solaris systems Raven is currently mostly binary-only because the Solaris kernel is unable to work with system libraries other than the ones matching exactly in version. Packages built on older releases of the OS work fine on newer ones, but for each OS release, a specific build environment would need to be created before building packages is possible. This is an issue that needs to be resolved in the future (I guess some help from the Illumos/Solaris community wouldn’t hurt). Also there are packages that don’t build on Solaris without patches which are not currently available. In case of important packages this leads to blockers since all other ports which depend on one such package also cannot be built: On FreeBSD there are 3,559 packages (including variants and metapackages) available from the repository at the present time. In the Solaris repo it’s only 2,851 packages. That’s certainly a nice start – but don’t expect to run a full-fledged desktop (or even X11 at all) there, yet!

In Linux land, distributions that come with glibc version 2.23 or newer work best. On distributions with older glibc versions (e.g. CentOS 7), software will not run as the standard C library is missing some required symbols. Raven will need to be bootstrapped again to support those distros. This is likely to happen before too long, but we’re not there, yet.

Current Firefox ESR version (+ sakura and pcmanfm in the panel)

MacOS (which might be supported soon), OpenBSD and NetBSD are not currently supported, nor is Linux with musl-libc or ╬╝clibc. Also currently Raven is amd64 only. ARM64 support is planned and i386 might or might not happen but are not available now.

Current status

At this time Raven is probably most interesting for people who love tech and enjoy tinkering on *nix systems as well as those who like the features and are ok with being early adopters. Yes, in general it’s ready for the latter. At least two people (including me) use Raven’s packages exclusively on one of their machines. I’d say it is ready as a daily driver (if you can live with the limited set of software available – or consider adding more ports). In fact I built a laptop that I use e.g. for on-call duty with it. Since that one is critical, it probably needs to be considered as “in production use”.

It’s possible to install various text mode applications with Raven, but X11 is also available. You can choose from multiple window managers or from at least two desktop environments (Lumina and the ultra-light EDE). Xfce4 is partially available (i.e. the panel is already ported). If you’re looking for web browsers, a current version of Firefox ESR (called “rustless-firefox”) can be installed as well as Surf, a simple webkit-based browser. The LibreOffice suite is available in its latest version, too. The same is true for the just released Perl 5.28 and Python 3.7.

Running Chocolate DooM and Chocolate Heretic

Oh, and if you’re into gaming… It’s not all just serious stuff. Yes, you can install and play DooM!

Conclusion

Ravenports is a fascinating project with lots and lots of possibilities. I wanted to get into porting with FreeBSD for quite a while but hesitated as I’m not a programmer. Then again I had been interested in package building for a long time and had played around with it on Arch Linux quite a bit. After my submissions to FreeBSD had been rotting in bug tracker for months (and still are after almost a year), I chose to give Raven a try in the meantime.

I was already familiar with Pkg and had used Synth before, too. Bootstrapping Raven’s pkg and then installing stuff was as easy as expected. The same was true for building the ports myself. Then I did quite a bit of reading and wrote my first port. It didn’t take more than 5 minutes after I opened my pull request on GitHub, before John responded – and the port was committed not much later. This was such a huge contrast that I decided to do more with Raven.

There was a learning curve, yes, but I received lots of help in getting started. I obviously liked the project enough to become a regular contributor and even got commit access to the ravensource repo later. Currently I’m maintaining just over 80 ports and I hope to write many more in the future. There have been some hard ports along the way (where I learned a lot about *nix), but lots of things are actually pretty easy once you get the hang of it.

Tongue-in-cheek: Make chaos or “make sense”!

If this post got you interested, just give it a try. Feel free to comment here and if you run into problems I’ll try to help. After this general overview of Raven the next post I plan to write will be on actually using it.