Rusted ravens: Ravenports march 2019 status update

It’s been a couple of months since I last wrote about Ravenports, the universal *nix application building framework. Not exactly being a slowly moving project, a lot has happened since then.

Platform support

Raven currently supports DragonFly BSD, FreeBSD, Linux and Solaris/Illumos, the latter being only in the form of binary packages (except for when you have access to an installation of Solaris 10u8 – which can be used to build packages, too).

People following the project will notice the lack of macOS/Darwin support mentioned here. This is not a mistake as support for that platform has been put on hold for now. While Raven has successfully been bootstrapped on macOS before, the developers have lost access to any macOS machines and thus cannot continue support for it.

This does not mean that platform is gone forever. It might be resurrected at a later point in time if given access to a Mac again. The adventurous can even try to bootstrap Raven on different platforms now as the process has been documented (with macOS as the example).

I intended to do some work on bootstrapping Raven on FreeBSD/ARM64 – only to find that FreeBSD unfortunately still has a long way before making that platform tier 1. At work I had access to server-class ARM64 hardware, but current versions of FreeBSD have trouble booting up and I could not get the network running at all (if you’re interested in the details see my previous post). I’m still hoping for reactions on the mailing list but until upstream FreeBSD is fixed on ThunderX trying to bootstrap does not make much sense.

Toolchain and package updates

The toolchain used by Ravenports has been updated to GCC 8.3 and Binutils 2.32 on all four supported platforms (access to Mac was lost before the toolchain update).

As usual, Solaris needed a bit of extra treatment but up to date compiler and tools are available for it now, too. Even better: The linker from the LLVM project (lld) is available and usable on Solaris/Illumos now as well. Since it takes several hours (!) to link on Solaris, a mostly static lld executable was added to the sysroot package for that platform. This way this long-building package does not have to be rebuilt as often.

Packages have been rebuilt with this bleeding-edge toolchain (plus the usual fallout has been collected and fixed). So if you are using Raven, you are making use of the latest compiler technology with the best in optimization. Of course a lot of effort went into providing the most current versions of the packaged software, too (at least where that is feasible).

On the desktop side of things I’ve added the awesome window manager to the list of available software. It’s actually my WM of choice, but not too many people are into tiling so I postponed this one for after making Xfce available. Work on bringing in more Lua-related ports for an advanced configuration it is ongoing, but the WM is already usable as it is now.

I’ve also done a bit of less visible work, going back to many ports that I created previously and added in missing license info. This work is also not completed, yet, but the situation is improving, of course.

Rust!

One of the big drawbacks of Ravenports as stated last time, was the lack of the Rust compiler. This effectively meant a showstopper for things like current versions of Firefox, Thunderbird, librsvg, etc. The great news is that this blocker has been mostly removed: Rust is available via Raven for Dragonfly, FreeBSD and Linux! Solaris/Illumos support is pending, I think that any helping hand would be greatly appreciated.

Bringing in Rust was a big project on its own. Adding an initial bootstrap package for Dragonfly alone took months (thank you, Mr. Neumann!). The first working version of the port made Rust 1.31 available. It has since been updated to version 1.32 and 1.33 and John has added functionality to the Raven framework to work with Rust’s crates as well as scripts to assist with future updates. Taking all of that into consideration, Rust support in Raven is already pretty good for the short time that we have it.

Eventually even a port for Firefox landed – as of now it’s marked broken, though. The reason is that while it does compile just fine, the program crashes when actually started. The exact cause for this is yet unknown. If anybody with some debugging abilities has a little time on his hands, nailing down what happens would be a task that a lot of people will be benefit from for sure!

Updated ravenadm

Ravenadm, the Ravenports administration tool, has seen several updates with new features. Some have brought internal changes or new features necessary for new or updated packages. One example is a project-wide fix for ports that build with Meson: Before the change many programs needed special treatment to make Meson honor the rpath for the resulting binaries. Now Raven can automatically take care of this, saving us a whole bunch of sed commands in the specification file. Another new feature is the “Solaris functions” mechanism which can automatically fix certain functions that required generating patches before. Certainly also very nice to have!

Probably my favorite new feature is that Ravenadm now supports concurrent processes in several cases: While you cannot start a second set of package builds at the same time for obvious reasons, it is now possible to ask Ravenadm in which bucket a certain port lives, sort manifests, and such while building packages! I cannot say how much the previous behavior got in my way while doing porting work… This makes porting much, much more pleasant.

A last improvement that I want to mention here is a rather simple one – however one that has a huge impact. Newer versions of Ravenadm put all license-related texts into the logs! This means you can simply look at the log and see if e.g. the terms got extracted correctly. Before you had to use the ENTERAFTER option to enter an interactive build session and look at the extracted file. This is a huge improvement for porters.

SSL

Another big and most likely unique feature added to Raven recently is SSL autoselection. Raven has had autoselection facilities for Python, Ruby and Perl for about a year now. The latter allow for multiple versions of the interpreters to be installed in parallel and take care of calling the actual binary with the same parameters, preferring the newest version over older ones (until configured differently).

Raven supports LibreSSL, OpenSSL as well as LibreSSL-devel and OpenSSL-devel. Before the change, you could select the SSL library to use in the profile and it would be used to link all packages against it. Now we have even more flexibility: You can e.g. build all the packages against LibreSSL by default and just fall back to OpenSSL for the few packages that really require it!

And in fact Raven takes it all one step further: You can have OpenSSL 1.0.2 and OpenSSL 1.1.1 (which introduced braking changes) installed in parallel and use packages on the same system where some require the new version and some that cannot use it, yet! Pretty nice, huh?

Future work

Of course there are still enough rough edges that require work. Probably the most pressing issue is to get Firefox working so Raven’s users can have access to a convenient and modern browser. There are also quite some programs which need ports created for them. The goal here is to provide the most critical programs to allow Dragonfly to make the switch from Dports to Ravenports for the official packages.

On FreeBSD Filezilla does not currently work: It cannot be built with GCC due to a compiler bug in GCC 7.x and 8.x. Therefore it is a special port that get’s build with Clang instead. The problem is that libfilezilla needs to be built with the same toolchain – and that cannot currently be built with Clang using Raven…

Raven on Linux has some packages not available due to additional dependencies on that platform. I begun adding some Linux-specific ports but lost motivation to do so pretty fast (there are enough other things after all). Also the package manager is still causing pain, randomly crashing.

Solaris is also missing quite some packages. This is due to additional patches being required for a lot of software to build properly. Ravenports tries to support this platform as good as possible; however this could surely be improved if anybody using Solaris or an Illumos distribution as his or her OS of choice would start using Raven and giving feedback or even contribute.

Get in touch!

Interested in Raven? Get in touch with us! There is an official IRC channel now (#ravenports on Freenode) which is probably the best place to talk to other Raven users or the porters and developers. You can of course also send an email.

If you want to contribute, there is now a new “Customsource” repository on GitHub that you can create pull requests against. Feel free to contribute anything from finished ports that might only need polish to WIP ports that are important for you but you got stuck with.

There are many other means of helping with the than project then doing porting work, though. Open issues if you find problems with packages or have an idea. Also tell us if you read the wiki and found something hard to understand. Or if you could use a tutorial for something – just ping me. Asking doesn’t hurt and chances are that I can write something up.

Got something else that I didn’t talk about here? Tell us anyway. We’re pretty approachable and less elitist than you might think people who work on new package systems would be! 😉

Advertisements

Ravenports explained: Why not just join XYZ?

As the year comes to an end, I’ve seen quite some interest in my previous post. There has been a question on Reddit what the benefit(s) of Raven over Pkgsrc might be and why the developers don’t simply join an existing effort instead of building something new.

I’ve touched on this topic about half a year ago, but I think the question is worth a detailed reply that fully covers both parts of it. So I’ll try to answer 1) why Ravenports exists in the first place and 2) what sets it apart from Pkgsrc and other ports systems.

Why maintain Ravenports instead of working on Pkgsrc?

Well, obviously because its author felt it was worthwhile to start and maintain the project! Of course that leads to another and more important question – why didn’t John Marino just join e.g. Pkgsrc instead? The answer to that is: Well… He did.

John got his NetBSD commit bit and became a Pkgsrc developer back in the day when DragonflyBSD still used Pkgsrc by default. He maintained a ton of ports there and made sure that other people’s ports still worked on DF after they had been updated. DragonflyBSD had been considered a first-class citizen by Pkgsrc. However there had been two big problems:

1) Being primarily a NetBSD project, Pkgsrc development takes place mostly on NetBSD of course. Things were tested on NetBSD and then committed. There was no testing done on the other supported platforms – which is a completely comprehensible decision given the amount of ports available and the number of supported platforms as well as the need to get software updated in a somewhat timely manner! However this lead to frequent breakage. A few suggestions that made sense from the Dragonfly perspective could not be agreed upon taking the whole of Pkgsrc into account. In the end the policy was: “If things in the tree break for your platform, go ahead and fix it.” So basically the answer to problem 1 was: “Throw more manpower at it.”

2) As the small project that DragonflyBSD is, there simply were not too many people available for this task however. In fact it was largely John alone who did most of the work with some help here and there. It’s impossible to spend resources that you don’t have available!

As you can see problem 1 causes problem 2 – and that one proved to be unfixable. Thus the problems with Pkgsrc grew and there was really not much that could have been done about it. And as the suggestions to somewhat relieve the worst impact were turned down, Dragonfly had to give up Pkgsrc. Please keep in mind that there’s a major difference between how Dragonfly used Pkgsrc and how some other platforms do. Sure, it’s great that you can use Pkgsrc on AIX to obtain some current software. Same thing for many other systems. Dragonfly used Pkgsrc just as NetBSD does, though: As the primary means to get software installed. Large-scale breakage of packages is a no-go in such a case, especially if it happens somewhat often and was bound to happen again and again.

Ok – another project then. Adapt the FPC maybe?

John then brought the new FreeBSD package manager as well as the FreeBSD ports collection over to Dragonfly with a system called “delta ports” or Dports. It’s basically an overlay with patches that Dfly requires to build those ports. Even though the FPC is meant for FreeBSD only and Pkgsrc – being cross-platform – might seem like the more logical candidate, this worked out a lot better and John maintained Dports for years.

In maintaining so many ports for both Pkgsrc and Dports he had a quite few ideas on how to do things better. They wouldn’t fit into the projects as they were organized, though. So he begun playing with various things on his own. Then… FreeBSD introduced flavored ports.

Don’t get me wrong here: I’m a FreeBSD user and I’m glad that flavored ports are finally available. However from a technical point of view they are implemented in a way that’s far from perfect. This is no wonder, though: When the ports tree was first introduced, nobody thought of flavors. What we have today is a fine example of a feature implemented as an afterthought. It works, yes, but it meant a disrupting change and broke expectations of all ports-related programs. It also made maintaining Dports much, much more time-intensive – to the point where it becomes no longer feasible to keep it up.

What does Ravenports have to offer over Pkgsrc?

Just like every younger project, Ravenports has the considerable advantage of starting fresh without the burden of choices that seemed right in the past but were probably regretted later. If this is combined with the will to learn from previous attempts to get packaging right as well as considerable experience with those, this has a lot of potential.

Think about it for a moment: FreeBSD’s ports collection shipped with the 1.0 release of the OS – and thus was created back in 1993. Pkgsrc began as a fork of it in 1997. So both were originally designed in a decade that has long passed (and in fact not even in this millennium!). Yes, both have been modernized over time. There are limits to this, however. It can be pretty hard to integrate new features into a structure that never meant to support anything like that. Do you think anybody in the mid 90’s could have thought about the needs of today? Ravenports deliberately does not support some old cruft. It’s meant for the coming decade of the 2020’s.

Here’s some strong points where Raven is ahead of Pkgsrc:

  • Tooling:
  • It offers a modern, integrated solution. There’s one control program (“ravenadm”) that deals with everything regarding Ravenports: It’s used to configure the package building system, it fetches the buildsheets (ports) and keeps them up to date, it builds all the packages or a subset thereof, …

  • Pristine package builds:
  • Everything is built in a chroot sandbox specifically assembled for that build process. There is no way that build dependencies clutter your build system (chances are you don’t want to use m4 or automake yourself and thus don’t need them installed on the OS). There’s also no way that installed packages of your system pollute the packages that Raven builds: The isolation prevents e.g. linking against additional stuff that you didn’t mean to.

  • It’s fast:
  • Did you ever run a bulk-build for Pkgsrc packages? Ravenports optimizes build times on modern systems by taking advantage of memory disks and such. The port scan alone makes a huge difference.

  • Potentially package manager agnostic:
  • Currently Raven supports only the Pkg package manager but as all it does is build packages, it was designed to support additional package managers if needed. You actually want it to generate rpm or pacman packages? Not currently implemented but certainly possible if desired.

  • Powerful default package manager:
  • Pkg, a modern tool for package management, is quite capable. If you read the manpages for it you will find out that it’s loaded with useful features. The old pkg_tools that Pkgsrc still use totally pale in comparison – and rightfully so.

  • Easy administration of multiple repos:
  • Need multiple repositories? No problem. Just create profiles for them. E.g. one that uses LibreSSL and another one that links against OpenSSL instead. Also you can choose the default version of Perl, Python, Ruby, ect. to use. And you can choose if MySQL should be Oracle’s MySQL, MariaDB, Galera, ect.

  • Convenient use of custom ports:
  • Can you use custom ports that are not in the official buildsheet collection? Sure thing. You can create directories for your custom ports and even use different ones in different profiles. Want to change an existing port? Just place one with the same name in your custom port directory and it will override the original one. Buildsheets from custom ports are generated automatically so there’s no hassle there. It probably doesn’t get much more convenient!

  • Variants and subpackages:
  • Package variants (i.e. “flavors”) and subpackages are not an afterthought and are thus used excessively right from the beginning. This makes package management with Raven very flexible.

  • Testing:
  • The Ravenports system has very strict rules for buildsheets. If the ravenadm tool considers a port to be valid, it is almost guaranteed that it is actually fine. Also packages can not only be mass-built but they can also be tested automatically as well (Is the RPATH ok? Are all required shared objects available? Is the manifest file complete? Are the required descriptions in place? Is the license ok or lacking? Things like that).

  • Automation:
  • Ravenports tries to automate many things that do not actually need human attention. For example quite often Python-related ports can be auto-generated. This saves time and effort of the maintainers that can be better spent on other things.

  • Modern day development:
  • Want to contribute something? It’s extremely easy. If you have a GitHub account you’re all set: Fork the git repo, make your changes, then commit and push them. Now all that’s left is opening a Pull Request. Yes, that’s all. If you don’t have a GH account, create one. Or send us patches as it was traditionally done. Ravenadm will happily create a template for you to assist you if you want to contribute a new port.

  • No ports ownership:
  • In Ravenports nobody “owns” a port. If you submitted one you become a contact for it. If somebody wants to make major changes to the port, that person is expected to contact you and communicate the proposals. Small or trivial changes however (like a simple version upgrade) can be done by anybody. This ensures rapid development and very fast adoption of new versions even if the original porter does not currently have the time to maintain everything in a timely manner.

  • Fast releases:
  • Ravensource provides new releases quite often. This way you can get pretty fresh software early on. There is no fixed time frame for it, though: Releases are made when it makes sense. If there have been major changes to the tree the next release might be delayed for testing.

  • Binary bootstrap:
  • Ravenports has a very simple and fast bootstrap process that makes use of binary packages for the respective platform. No system compiler required! Raven brings in its own full toolchain.

There are of course cases where it makes sense to use Pkgsrc and it’s not too hard to find any: E.g. if you need packages for a platform that’s unsupported in Raven or if you need software not yet available there. In the end this is Open Source: We’re all friends and using the right tool for the job makes sense.

Couldn’t Ports/Pkgsrc be modernized?

I’ve used Pkgsrc both in private and at work and I’m pretty happy that it’s available when I need it. But I don’t like the old pkg_tools much. They do their job but they are far from modern programs and really feel like relics today. And while I’m pretty happy with FreeBSD’s ports, those aren’t portable (and for some reason I’ve never been completely happy with Poudriere, FreeBSD’s package builder).

Before finally creating Ravenports, John wrote Synth, a very nice package builder for FreeBSD and DragonflyBSD that supports Ports/Dports. It has been put on hold in favor of Raven, but it is still maintained and I continue to use it on FreeBSD to build my packages.

John also created Pkgsrc-synth. It’s a version of Pkgsrc that uses the Pkg package manager. I’ve never tried it out – but it was stopped exactly two month ago as there seems to not have been any interest from the Pkgsrc people. I think this is a pitty, as pkg is really nice and has the right license for any BSD project. It could have been a chance to move Pkgsrc into a more modern direction. But meh.

Conclusion

Raven does not exist because everything else sucks. It exists because all the other candidates proved to not quite fit the needs of Ravenport’s author. As such it is a chance to keep the good parts of its various precursors that it heavily draws inspiration from. It’s a chance to combine these good parts to make something awesome. And it’s a chance to implement a lot of new ideas that should make sense in modern-day *nix package building which – for various reasons – cannot have a place in the old projects.

There’s still a lot of work to do, but we’re getting there. In my previous post I wrote that one of the big shortcomings was the lack of Rust. In the meantime Rust support has landed for DragonflyBSD, FreeBSD and Linux.

If there are any more questions feel free to post them here. I’m not on Reddit and I just saw the above question by accident. So I cannot promise to answer anywhere else than here.

Happy new year everyone!

One year of flying with the Raven: Ready for the Desktop?

It has been a little over one year now that I’m with the Ravenports project. Time to reflect my involvement, my expectations and hopes.

Ravenports

Ravenports is a universal packaging framework for *nix operating systems. For the user it provides easy access to binary packages of common software for multiple platforms. It has been the long-lasting champion on Repology’s top 10 repositories regarding package freshness (rarely dropping below 96 percent while all other projects keep below 90!).

For the porter it offers a well-designed and elegant means of writing cross-platform buildsheets that allow building the same version of the software with (completely or mostly) the same compile-time configuration on different operating systems or distributions.

And for the developer it means a real-world project that’s written in modern Ada (ravenadm) and C (pkg) – as well as some Perl for support scripts and make. Things feel very optimized and fast. Not being a programmer though, I cannot really say anything about the actual code and thus leave it to the interested reader’s judgement.

If you’re interested in a more comprehensive introduction to Ravenports, I’ve written one half a year ago.

Platforms

Ravenports has initially been developed on DragonFly BSD. When I became aware of it, it had already been ported to work on Linux, too. I liked the idea of the project, but had no DF or Linux boxes available for tinkering and didn’t feel like setting one up. Thus I moved on.

As I checked back a little later, FreeBSD support had been added. Since I had just lost my excuse not to try it out right away, I started playing with it – and was pretty happy. At that time I had trouble to get a port that I wrote into FreeBSD’s Ports Collection and thought that Raven could be an excellent playground to learn something and get a bit of experience that might help me later with FreeBSD.

The Xfce4 desktop – installed via Raven

I’ve long changed my mind, though! Raven is rather similar to FreeBSD’s ports system in many ways but where it differs it’s clearly superior. Also I love the cross-platform aspect and thus Raven is simply the better place for me to make home.

This year saw the introduction of Solaris/Illumos support that I tried out on OmniOS. Also Darwin support landed, upping the count of supported platforms to 5 already! Not too bad for a young project, huh? While Raven does work on all five platforms now it does so to varying degrees. But more on that later.

General activity

The Ravenports project consists of multiple Git repositories hosted on GitHub. The first one is Ravensource which most importantly holds the “raw” ports as they are written by the porters. It’s the most busy repo with over 5.200 commits since March 2017 (including almost 500 by me).

Then there’s the actual Ravenports repo that mostly contains the buildsheets which are compiled from Ravensource. It has over 1.400 commits right now.

Installing the xfce-single-core meta-package

Finally there’s the repo for the Ravenadm command-line tool. It’s approaching 900 commits since February 2017.

There’s still more to Raven like the Pkg package manager from FreeBSD (that was modified to add Zstd compression support) or libbsd4sol, a portability library which allows building code on Solaris that uses BSDisms (which was needed to add support for that platform to Raven). Most of the work on all repos was done by John alone.

With over 100 pull requests and more than 20 issues it’s clear now that there’s some interest in the project. Raven is still very small, though, with 6 people haveing contributed ports so far. After learning the basics and opening pull requests for half a year, I’ve been granted write-access to the source repository. Just recently I was able to push my 100th active port (there have been ports that became obsolete and were removed).

In general I’d say that there could of course be more people around and that the project would benefit from being able to provide more packages – though more than 3.200 is not bad at all! Also it’s good that there seems to be a growing user base which is even more important than having more porters join in. From my point of view, Raven is a healthy and fast-moving project. Still young, but doing well and heading in the right direction.

Major changes

There have been some pretty big changes that happened with Raven over time. Initially John started with a GCC6-based toolchain, only to switch to GCC7 when that was released. That was before my time with the project, but I witnessed the switch to GCC8.

Changing the toolchain certainly is a major interruption and most people are advised to just wait for the official repository to be re-rolled and then update. I had some bad luck in this regard – literally the day after I finally completed a working (and almost complete) set of basic packages for the FreeBSD_i386 platform, I faced the change to GCC8. Due to a lack of time I still haven’t repeated the switch on i386 (but I still plan to do it sometime).

The thunar file manager

Other changes that always have a huge impact (causing lots and lots of packages to be rebuilt) is adopting a new version (as well as dropping an old one) of the popular interpreter languages like Python, Perl and Ruby. Ravenports always supports two versions of Perl and Ruby and two versions of Python 3 (as well as 2.7 for now). So when Python 3.7 was released, 3.5 was removed and Perl 5.24 had to go when 5.28 was added.

Recently the former LLVM port that included everything regarding LLVM was split (LLVM, Clang, lld, openmp). Also now and then new statements are added to Ravenadm, so that old versions cannot work with a new release of the buildsheet repository (which is called “conspiracy”). But this is pretty easy to work around compared to the changes mentioned before.

So on the whole, Raven has proven that it can easily stand even big changes. For me this is essential to build faith in a project. And Raven is doing well in this regard.

Desktop-ready?

There are lots of people who will want to use Raven on servers. That’s totally fine of course. But for a project as ambitious as Ravenports, it’s necessary to provide a somewhat comfortable environment for the developers and the users alike. If it doesn’t manage to become a daily driver for people it cannot succeed.

For that reason I decided to work towards good desktop support for the little dev machine that I dedicated to my work on the project. When I started, X11 was already working and Openbox had freshly landed in the repos. So I had a simplistic environment to work with: Openbox + Xterm. However I could not even change my keyboard layout! Therefor I wrote a port for setxkbmap and eventually it was accepted as the first outside contribution to the project.

The Surf web browser

Next I did some work to get the FLTK toolkit and the EDE desktop in. Then I added my favorite terminal emulator, Sakura. This worked out pretty well and the biggest shortcoming at the end of 2017 was that there was no real graphical browser available. A lot has changed since then!

Desktop choices

Today you can choose between multiple window managers, both floating and tiling:

  • twm
  • cwm
  • openbox
  • fluxbox
  • xfwm4
  • pekwm
  • i3

And in case you prefer a real desktop environment, there are also several available:

  • Lumina (moderate, Qt-based)
  • Xfce4 (somewhat light-weight, GTK-based)
  • EDE (extremely frugal and minimalistic, FLTK-based)

Two graphical web browsers are available, Surf (which is deliberately simplistic and does not even support tabs) as well as an old version of Firefox (the last one that builds without Rust). This is certainly not perfect but much better than a year before.

Also other important programs are available, including LibreOffice! Last month the Apache webserver landed – which is a pretty complex port compared to many others.

Shortcomings

Are there packages you’ll miss? Most certainly. However there’s a wishlist now with ports that people would like to see created (please feel free to add more requests there). And that’s another good step ahead. Currently it’s almost 120 items long. Fortunately there’s been some success, too, and 26 requested ports have been created and taken of the list so far.

There are some future ports that will require lots of effort (hint: Help wanted!). The most important one that blocks some other important ports is the Rust compiler. There has been some work done on this but it’s not done, yet. Another real beast is TeX. This totally must be supported at some point. Current versions of Firefox and Chromium are often asked for. And somebody even requested Eclipse (which needs Java!). So there’s definitely more than enough work to do.

Using Raven on Linux works, but there are some flaws. Initially the Pkg package manager used to crash quite often. John traced that back to a bug in the version of SQlite that’s used internally by Pkg: The problem only struck on Linux and was fixed by using a newer version instead. While it’s much better now, there’s still the occasional problem with it.

While the packages from the repo work finde on Solaris 10u8 and above as well als Illumos, the exact version 10u8 is currently required to build packages. This is due to Solaris not being able to work with older system libraries in the build chroot. It would be great to haven an alternative ravensys-root for any Illumos distribution (OmniOS, SmartOS, Tribblix, …) available so that interested people without access to that specific closed-source Solaris version can develop Raven on that platform.

I don’t know how well Raven works on Darwin. Since I don’t have access to any macOS machines and PureDarwin is not really ready, yet, there’s currently no chance for me to test it. I intend to buy an older MacBook or something in the future, though, if I come across a fair offer and have some money available to spend on my hobby.

Some ports are not available on one platform or the other: Illumos mostly because they’d require patches to build and Linux often because it relies on additional libraries that have not yet been added to Raven. And then there’s a lot of packages that are mostly untested. All of these issues can be fixed, of course. All of those require a larger user-base, though. So it’s probably the best strategy to keep working on making Raven attractive to more users and address things when the right people show up.

What’s to come?

Currently Raven uses the primordial X11 input drivers (xf86-input-keyboard and xf86-input-mouse) on all platforms. In 2013 Linux pioneered support for generic input drivers by exposing the kernels “event devices”. Not too much later many Linux distributions adopted xf86-input-evdev. In 2014 there was a GSOC project to add evdev support for FreeBSD. Like many projects it came along a good part of the way but eventually was left unfinished. It was picked up and completed by a FreeBSD developer in 2016.

Xfce’s settings and applications menu

To use it, a special kernel had to be built so it would expose /dev/input device nodes. Then a sysctl had to be set – and eventually X11 had to be patched for emulated udev support… Why would anybody want to do all this just for different input drivers? Multi-touch support is just one valid reason. Another one is that having evdev-based input drivers is half the way to eventually support libinput, too. And that is one of the prerequisites for Wayland!

This month FreeBSD has finally enabled evdev support in the GENERIC kernel in both -CURRENT and 12-STABLE. That means the upcoming FreeBSD 12.0 will not support it out of the box, but most likely a future 12.1 will. Dragonfly BSD has also grown support for event devices and people are interested in working towards Wayland. I hope that we’ll be able to get xf86-input-evdev working with our X11 (on Dragonfly, FreeBSD and Linux) next year,

I’m taking a little break from Xfce now (but plan to port most of the remaining components later to make it a well-supported DE in Raven). There are a few things I have planned like adding Linux support for OpenVPN (it depends on some libraries and programs that are Linux only which are not yet in Raven). Also I intend to take a look at adding some more Qt5 components and write a few requested ports. And finally I want to write another post next year – a tutorial on using Ravenports and creating new ports.

So keep flying with us – it’s exciting times!

Exploring OmniOS in a VM (2/2)

This is the second part of my post about “exploring OmniOS in a VM”. The first post showed my adventures with service and user management on a fresh installation. My initial goal was to make the system let me ssh into it: Now the SSH daemon is listening and I’ve created an unprivileged user. So the only thing still missing is bringing up the network to connect to the system from outside.

Network interfaces

Networking can be complicated, but I have rather modest requirements here (make use of DHCP) – so that should not be too much of a problem, right? The basics are pretty much the same on all Unices that I’ve come across so far (even if Linux is moving away from ifconfig with their strange ip utility). I was curious to see how the Solaris-derived systems call the NIC – but it probably couldn’t be any worse than enp2s0 or something that is common with Linux these days…

# ifconfig
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
          inet 127.0.0.1 netmask ff000000
lo0: flags=200200849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
          inet6 ::1/128

Huh? Only two entries for lo0 (IPv4 and v6)? Strange! First thought: Could it be that the type of NIC emulated by VirtualBox is not supported? I looked for that info on the net – and the opposite is true: The default VirtualBox NIC (Intel PRO/1000) is supported, the older models that VBox offers aren’t!

Ifconfig output and fragment of the corresponding man page

Obviously it’s time again to dig into manpages. Fortunately there’s a long SEE ALSO section again with ifconfig(1M). And I’ve already learned something so far: *adm commands are a good candidate to be read first. Cfgadm doesn’t seem to be what I’m looking for, but dladm looks promising – the short description reads: “Administer data links”.

I must say that I like the way the parameters are named. No cryptic and hard to remember stuff here that makes you wonder what it actually means. Parameters like “show-phys” and “show-link” immediately give you an idea of what they do. And I’m pretty sure: With a little bit of practice it will work out well to guess parameters that you’ve not come across, yet. Nice!

# dladm show-link
LINK         CLASS     MTU    STATE     BRIDGE     OVER
e1000g0      phys      1500   unknown   --         --

Ok, there we have the interface name: e1000g0. I don’t want to create bridges, bond together NICs or anything, so I’m done with this tool. But what to do with that information?

The ifconfig manpage mentions the command ipadm – but it’s kind of hidden in the long text. For whatever reason it’s missing in SEE ALSO! I’d definitely suggest that it’d be added, too. More often than not people don’t have the time to really read through a long manpage (or are not as curious about a new system as I am and thus don’t feel like reading more than they have to). But anyways.

Ipadm manpage

# ipadm create-if e1000g0

This should attach the driver and create the actual interface. And yes, now ifconfig can show it:

Interface e1000g0 created

Network connection?

Almost there! Now the interface needs an IP address. Looks like ipadm is the right tool for this job as well. I had some trouble to find out what an interface-id is, though. The manpage obviously assumes the reader is familiar with that term, which was not the case with me. I tried to find out more about it, but was unable to locate any useful info in other manpages. So I resorted to the OmniOS wiki again and that helped (it seems that you can actually choose almost anything as an Interface ID but there are certain conventions). Ok, let’s try it out and see if it works:

# ipadm create-addr -T dhcp e1000g0/v4

No error or anything, so now the system should have acquired an IPv4 address.

IPv4 address assigned via DHCP

10.0.2.15, great! Let’s see if we can reach the internet:

# ping elderlinux.org
ping: unknown host elderlinux.org

DNS pt. 1

Ok, looks like we don’t have name resolution, yet. Is the right nameserver configured?

# cat /etc/resolv.conf
cat: cannot open /etc/resolv.conf: No such file or directory

Oops. There’s something not right here! If I configure my FreeBSD box to use DHCP that makes sure resolv.conf is populated properly. The interface e1000g0 got an IP – so the DHCP request must have been successful. But did something break before the network was configured completely? Is the DHCP client daemon even running?

DHCP

# ps aux | grep -i [d]hcp
root       551  0.0  0.1 2836 1640 ?        S 18:52:22  0:00 /sbin/dhcpagent

Hm! I’m used to dhclient, but dhcpagent is unknown to me. According to the manpage, it’s the DHCP client daemon, so that must be what OmniOS uses to initiate and renew DHCP requests. And obviously it’s running. However the manpage solves the mystery:

Aside from the IP address, and for IPv4 alone, the netmask, broadcast address, and the default router, the agent does not directly configure the workstation, but instead acts as a database which may be interrogated by other programs, and in particular by dhcpinfo(1).

Ah-ha! So I have to configure things like DNS myself. Nevertheless the system should have received the correct nameserver IP. Let’s see if we can actually get it via the dhcpinfo command that I’ve just learned about. A look at the manpage as well as at the DHCP inittab later I know how to ask for that information:

# dhcpinfo -i e1000g0 DNSserv
192.168.2.1

Right, that’s my nameserver.

Some lines from /etc/dhcp/inittab

Route?

Is the routing information correct? Let’s check.

# netstat -r -f inet
[...]
default      10.0.2.2  [...]
10.0.2.0     10.0.2.15 [...]
localhost    localhost [...]

Looks good and things should work. One more test:

# route get 192.168.2.1
   route to: fw.local
destination: default
       mask: default
    gateway: 10.0.2.2
  interface: e1000g0
[...]

Alright, then it’s probably not a network issue… But what could it be?

DNS pt. 2

Eventually I found another hint at the wiki. Looks like by default OmniOS has a very… well, old-school setting when it comes to sources for name resolution: Only the /etc/hosts is used by default! I haven’t messed with nsswitch.conf for quite a while, but in this case it’s the solution to this little mystery.

# fgrep "hosts:" /etc/nsswitch.conf
hosts:      files

There are a couple of example configurations that can be used, though:

# ls -1 /etc/nsswitch.*
/etc/nsswitch.ad
/etc/nsswitch.conf
/etc/nsswitch.dns
/etc/nsswitch.files
/etc/nsswitch.ldap
/etc/nsswitch.nis

Copying nsswitch.dns over nsswitch.conf should fix that problem.

And really: Instead of getting a ping: unknown host elderlinux.org now I get no answer from elderlinux.org – which is OK, since my nameserver doesn’t answer ping requests.

Remote access

Now with the network set up correctly, it’s finally time to connect to the VM remotely. To be able to do so, I configure VirtualBox to forward port 22 of the VM to port 10022 on the host machine. And then it’s time to try to connect – and yes, it works!

SSHing into the OmniOS VM… finally!

Conclusion

So much for my first adventure with OmniOS. I had quite a few difficulties to overcome even in this very simple scenario of just SSHing into the VM. But what could have been a trivial task proved to be rather educational. And being a curious person, I actually enjoyed it.

I’ll take a break from the Illumos universe for now, but I definitely plan to visit OmniOS again. Then I’ll do an installation on real hardware and plan to take a look at package management and other areas. Hope you enjoyed reading these articles, too!

Exploring OmniOS in a VM (1/2)

While I’ve been using Unix-like systems for quite a while and heavily for about a decade, I’m completely new to operating systems of the Solaris/Illumos family. So this post might be of interest for other *BSD or Linux users who want to take a peek at an Illumos distribution – or to the Illumos people interested in how accessible their system is for users coming from other *nix operating systems.

In my previous post I wrote about why I wanted to take a look at OmniOS in the first place. I also detailed the installation process. This post is about the first steps that I took on my newly installed OmniOS system. There’s a lot to discover – more actually than I first thought. For that reason I decided to split what was meant to be one post into two. The first part covers service management and creating a user.

Virtualize or not?

According to a comment, a Reddit user would be more interested in an installation on real hardware. There are two reasons why I like to try things out using a hypervisor: 1) It’s a quick thing to do and no spare hardware is required 2) it’s pretty easy to create screenshots for an article like this.

OmniOS booted up and root logged in

However I see the point in trying things out on real hardware and I’m always open to suggestions. For that reason I’ll be writing a third post about OmniOS after this one – and will install it on real hardware. I’ve already set a somewhat modern system aside so that I can test if things work with UEFI and so on.

Fresh system: Where to go?

In the default installation I can simply login as root with a blank password. So here we are on a new OmniOS system. What shall we explore first?

The installer asked me which keymap to use and I chose German. While I’m familiar with both the DE and US keymaps enough that I can work with them, I’ve also learned an ergonomic variant called Neo² (think dvorak or colemak on steroids) and strongly prefer that.

It’s supported by X11 and after a simple setxkbmap de neo -option everything is fine. On the console however… For FreeBSD there’s a keymap available, but for Illumos-based systems it seems I’m out of luck.

So here’s the plan: Configure the system to let me SSH into it! Then I can use whatever I want on my client PC. Also this scenario touches upon a nice set of areas: System services, users, network… That should make a nice start.

Manpages (pt. 1)

During the first startup, smf(5) was mentioned so it might be a good idea to look that up. So that’s the service management facility. But hey, what’s that? The manpage clearly does not describe any type of configuration file. And actually the category headline is “Standards, Environments, and Macros”! What’s happening here?

smf(5) manpage

First discovery: The manpage sections are different. Way different, actually! Sections like man1, man1b, man1c, man3, man3bsm, man3contract, man3pam, etc… Just take a look. Very unfamiliar but obviously clearly arranged.

The smf manpage is also pretty informative and comprehensive including the valuable see also section. The same is true for other pages that I’ve looked at so far. On the whole this left a good impression on me.

System services

Solaris has replaced the traditional init system with smf and Illumos inherited it. It does more than the old init did, though: Services are now supervised, too. If a service is meant to be kept running, smf can take care of restarting it, should it die. It could be compared to systemd in some regards, but smf was created earlier (and doesn’t have the same … controversial reputation).

Services are described by a Fault Management Resource Identifier or FMRI which looks like svn:/network/loopback:default but can be shortened if they are unambiguous. I had no idea how to work with all this but the first “see also” reference was already helpful: scvs can be used to view service states (and thus get an idea about what the system is doing anyway).

svcs: service status

Another command caught my attention right away, too: svcadm sounded like it might be useful. And indeed this was just what I was searching for! The manpage revealed a really straight-forward syntax, so here we go:

# svcadm enable sshd
svcadm: Pattern 'sshd' doesn't match any instances
# svcadm enable ssh

The latter command did not produce any output. And since we’re in Unix territory here, that’s just what you want: It’s the normal case that something worked as intended, no need for output. Issuing a svcs and grepping for ssh shows it now so it should be running. But is SSH really ready now? Let’s check:

# sockstat -4 -l
-bash: sockstat: commmand not found

Yes, right, this is not FreeBSD. One netstat -tulpen later I know that it’s not exactly like Linux, either. Once more: man to the rescue!

# netstat -af inet -P tcp

The output (see image below) looks good. Let’s do one last check and simply connect to the default SSH port on the VM: Success. SSHd is definitely running and accepting connections.

Testing if the SSH daemon is running

Alright, so much for very basic service management! It’s just a test box, but I don’t really like sshing into it as root. Time to add a user to the system. How hard could that be?

User management

Ok, there’s no user database like on FreeBSD or anything like that. It’s just plain old /etc/passwd and /etc/shadow. How boring. So let’s just create a user and go on with the next topic, right?

# useradd -m kraileth
UX: useradd: ERROR: Unable to create home directory: Operation not applicable.

Uh oh… What is this? Maybe it’s not going to be so simple after all. And really: Reading through the manpage for useradd, I get a feeling that this topic is everything – but certainly not boring!

There’s a third file, /etc/user_attr, involved in user management. Users and roles can be assigned extended attributes there. Users can have authorizations, roles, profiles and be part of a project. I won’t go into any detail here (it makes sense to actually read the fine manpages yourself if you’re interested). Now if you’re thinking that some of this might be related to what FreeBSD does with login.conf, you’re on the right track. I cannot claim that I understood everything after reading about it just once. But it is sufficient to get an idea of what sort of complex and fine-grained user management Illumos can do!

Content of /etc/user_attr

Manpages (pt. 2)

Ok, while this has been an interesting read and is certainly good to know, it didn’t solve the problem that I had. The useradd manpage even has a section called DIAGNOSTICS where several possible errors are explained – however the one that I’m having isn’t. And that’s a pitty, since some of the ones listed here are pretty self-explanatory while “Operation not applicable” isn’t (at least for me).

I read a bit more but didn’t find any clue to what’s going on here. When my man skills failed me I turned to documentation on the net. And what better place to start with than the OmniOSce Wiki?

When it comes to local users, the first sentence (ignoring the missing word) reads: On OmniOS, home is under automounter control, so the directory is not writable.

Ah-ha! That sounds quite a bit like the reason for the mysterious “not applicable”! That should be hint enough so I can go on with manpages, right?

# apropos automounter
/usr/share/man/whatis: No such file or directory
# man -k automounter
/usr/share/man/whatis: No such file or directory

Hm… Looks like the whatis database does not exist! Let’s create it and try again:

# man -w
# apropos automounter
# apropos automount
autofs(4)      - automount configuration properties
automount(1m)  - install automatic mount points
automountd(1m) - autofs mount/unmount daemon

There we go, more pages to read and understand.

User’s home directories

The automounter is another slightly complex topic (and the fact that the wiki mentions /export/home while useradd seems to default to /home doesn’t help to prevent confusion, either). So I’m going to sum up what I found out so far:

It seems that people at Sun had the idea that it would be nice to be able work not only on your own workstation but at any other one, too. Managing users locally on each machine would be a nightmare (with people coming and going). Therefore they created the Yellow Pages, later renamed to NIS (Network Information Service). If you have never heard of it, think LDAP (as that has more or less completely replaced NIS today). Thus it was possible to get user information over the net instead of from local passwd and friends.

The logical next step was shared home directories so employees could have fully networked user logins on multiple machines. Sun already had NFS (Network File System) which could be used for the job. But it made sense to accompany it with the automounter. So this is the very short story of why home directories are typically located in /export/home on Solaris-derived operating systems: They were meant to be shared via NFS!

So we have to add a line to /etc/auto_home to let the automounter know to handle our new home directory:

* localhost:/export/home/&

Configuring the automounter for the home directory

Most of the two automounter configuration files are made up of the CDDL (pronounced “cuddle”) license – I’ve left it out here by using tail (see picture). After adding the needed rule (following the /export/home standard even though I don’t plan on using shared home directories), the autofs daemon needs to be restarted, then the user can finally be created:

# mkdir /export/home
# useradd -m -b /export/home kraileth
# passwd kraileth

Creating a new user

So with the user present on the system, we should be able to SSH into the local machine with that user:

ssh kraileth@localhost

SSHing into the system locally

Success! Now all that remains is bringing the net up.

What’s next?

The next post will be mostly network related and feature a conclusion of my first impressions. I hope to finish it next week.

A look beyond the BSD teacup: OmniOS installation

Five years ago I wrote a post about taking a look beyond the Linux teacup. I was an Arch Linux user back then and since there were projects like ArchBSD (called PacBSD today) and Arch Hurd, I decided to take a look at and write about them.

Things have changed. Today I’m a happy FreeBSD user, but it’s time again to take a look beyond the teacup of operating systems that I’m familiar with.

Why Illumos / OmniOS?

There are a couple of reasons. The Solaris derivatives are the other big community in the *nix family besides Linux and the BSDs and we hadn’t met so far. Working with ZFS on FreeBSD, I now and then I read messages that contain a reference to Illumos which certainly helps to keep up the awareness. Of course there has also been a bit of curiosity – what might the OS be like that grew ZFS?

Also the Ravenports project that I participate in planned to support Solaris/Illumos right from the beginning. I wanted to at least be somewhat “prepared” when support for that platform would finally land. So I did a little research on the various derivatives available and settled on the one that I had heard a talk about at last year’s conference of the German Unix Users Group: “OmniOS – Solaris for the Rest of Us”. I would have chosen SmartOS as I admire what Bryan Cantrill does but for getting to know Illumos I prefer a traditional installation over a run-from-RAM system.

There was also a meme about FreeBSD that got me thinking:

Internet Meme: Making fun of FreeBSD

Of course FreeBSD is not run by corporations, especially when compared to the state of Linux. And when it comes to sponsoring, OpenBSD also takes the money… When it comes to FreeBSD developers, there’s probably some truth to the claim that some of them are using macOS as their desktop systems while OpenBSD devs are more likely to develop on their OS of choice. But then there’s the statement that “every innovation in the past decade comes from Solaris”. Bhyve alone proves this wrong. But let’s be honest: Two of the major technologies that make FreeBSD a great platform today – ZFS and DTrace – actually do come from Solaris. PAM originates there and a more modern way of managing services as well. Also you hear good things about their zones and a lot of small utilities in general.

In the end it was a lack of time that made me cheat and go down the easiest road: Create a Vagrantfile and just pull a VM image of the net that someone else had prepared… This worked to just make sure that the Raven packages work on OmniOS. I was determined to return, though – someday. You know how things go: “someday” is a pretty common alias for “probably never, actually.”

But then I heard about a forum post on the BSDNow! podcast. The title “Initial OmniOS impressions by a BSD user” caught my attention. I read that it was written by somebody who had used FreeBSD for years but loathed the new Code of Conduct enough to leave. I also oppose the Conduct and have made that pretty clear in my February post [ ! -z ${COC} ] && exit 1. As stated there, I have stayed with my favorite OS and continue to advocate it. I decided to stop reading the post and try things out on my own instead. Now I’ve finally found the time to do so.

First attempt at installing the OS

OmniOS offers images for three branches: Stable, LTS and Bloody. Stable releases are made available twice a year with every fourth release being supported for three years (LTS) instead of one. Bloody images are more or less development snapshots meant for advanced users who want to test the newest features.

I downloaded the latest stable ISO and spun up a VM in Virtual Box. This is how things went:

Familiar Boot Loader

Ah, the good old beastie menu – with some nice ASCII artwork! OmniOS used GRUB before but not too long ago, the FreeBSD loader was ported over to Illumos. A good choice!

Two installers available

It looks like the team has created a new installer. I’m a curious person and want to know what it was like before – so I went with the old text-based installer.

Text installer: Keymap selection

Not much of a surprise: The first thing to do is selecting the right keymap.

ZFS pool creation options

Ok, next it’s time to create the ZFS pool for the operating system to install on. It seems like the Illumos term is rpool (resource pool I guess?). Since I’m just exploring the OS for the first time, I picked option 1 and… Nothing happened! Well, that’s not exactly true, since a message appears for a fraction of a second. If I press 1 again, it blinks up briefly again. Hm!

I kept the key pressed and try my best to read what it’s saying:
/kayak/installer/kayak-menu[254]: /kayak/installer/find-and-install: not found [No such file or directory]

Oops! Looks like there’s something broken on the current install media… So this was a dead-end pretty early on. However since we’re all friends in Open Source, I filed an issue with OmniOS’s kayak installer. A developer responded the next day and the issue was solved. This left a very good impression on me. Quality in development doesn’t show in that you never introduce bugs (which is nearly impossible even for really lame programs) but in how you react to bugs being found. Two thumbs up for OmniOS here (my latest PRs with FreeBSD have been rotting for about a year now)!

Dialog-based installer

What a great opportunity to test the new installer as well! Will it work?

Dialog-based installer: Keymap selection

Back on track with the dialog-based installer. Keymap selection is now done via a simple menu.

ZFS pool creation options

Ok, here we are again! Pool creation time. In the new installer it just does its job…

Disk selection

… and finds the drives, giving me a choice on where to install to. Of course it’s a pretty easy decision to make in case of my VM with just one virtual drive!

ZFS Root Pool Configuration

Next the installer allows for setting a few options regarding the pool. It’s nice to see that UEFI seems to be already supported. For this VM I went with BIOS GPT, though.

Hostname selection

Then the hostname is set. For the impatient (and uncreative) it suggests omniosce. There’s not too much any installer could do to set itself apart (and no need to).

Time zone selection 1

Another important system configuration is time zone settings. Since there’s a lot of time zones, it makes sense to group them together by continent instead of providing one large list. This looks pretty familiar from other OS installations.

Time zone selection 2

The next menu allows for selecting the actual time zone.

Time zone confirmation

Ok, a confirmation screen. A chance to review your settings probably doesn’t hurt.

Actual copying of OS data

Alright! Now the actual installation of the files to the pool starts.

Installer: Success!

Just a moment later the installation is finished. Cool, it even created a boot environment on its own! Good to see that they are so tightly integrated into the system.

Last step

Finally it’s time to reboot. The installer is offering to do some basic configuration of the system in case you want to do that.

Basic configuration options

I decided not to do it as you probably learn most when you force yourself to figure out how to configure stuff yourself. Of course I was curious, though, and took a peek at it. If you choose to create a user (just did this in another VM, so I can actually write about the installer), you’ll get to decide if you want to make it an administrative user, whether to give it sudo privileges and if you want to allow passwordless sudo. Nice!

First start: Preparing services

After rebooting the string “Loading unix…” made me smile and I was very curious about what’s to come. On the first boot it takes a bit longer since the service descriptions need to be parsed once. It’s not a terribly long delay, though.

First login on the new system

And there we have it, my first login into an OmniOS system installed myself.

What’s next?

That’s it for part one. In part two I’ll try to make the system useful. So far I have run into a problem that I haven’t been able to solve. But I have some time now to figure things out for the next post. Let’s see if I manage to get it working or if I have to report failure!