One year of flying with the Raven: Ready for the Desktop?

It has been a little over one year now that I’m with the Ravenports project. Time to reflect my involvement, my expectations and hopes.

Ravenports

Ravenports is a universal packaging framework for *nix operating systems. For the user it provides easy access to binary packages of common software for multiple platforms. It has been the long-lasting champion on Repology’s top 10 repositories regarding package freshness (rarely dropping below 96 percent while all other projects keep below 90!).

For the porter it offers a well-designed and elegant means of writing cross-platform buildsheets that allow building the same version of the software with (completely or mostly) the same compile-time configuration on different operating systems or distributions.

And for the developer it means a real-world project that’s written in modern Ada (ravenadm) and C (pkg) – as well as some Perl for support scripts and make. Things feel very optimized and fast. Not being a programmer though, I cannot really say anything about the actual code and thus leave it to the interested reader’s judgement.

If you’re interested in a more comprehensive introduction to Ravenports, I’ve written one half a year ago.

Platforms

Ravenports has initially been developed on DragonFly BSD. When I became aware of it, it had already been ported to work on Linux, too. I liked the idea of the project, but had no DF or Linux boxes available for tinkering and didn’t feel like setting one up. Thus I moved on.

As I checked back a little later, FreeBSD support had been added. Since I had just lost my excuse not to try it out right away, I started playing with it – and was pretty happy. At that time I had trouble to get a port that I wrote into FreeBSD’s Ports Collection and thought that Raven could be an excellent playground to learn something and get a bit of experience that might help me later with FreeBSD.

The Xfce4 desktop – installed via Raven

I’ve long changed my mind, though! Raven is rather similar to FreeBSD’s ports system in many ways but where it differs it’s clearly superior. Also I love the cross-platform aspect and thus Raven is simply the better place for me to make home.

This year saw the introduction of Solaris/Illumos support that I tried out on OmniOS. Also Darwin support landed, upping the count of supported platforms to 5 already! Not too bad for a young project, huh? While Raven does work on all five platforms now it does so to varying degrees. But more on that later.

General activity

The Ravenports project consists of multiple Git repositories hosted on GitHub. The first one is Ravensource which most importantly holds the “raw” ports as they are written by the porters. It’s the most busy repo with over 5.200 commits since March 2017 (including almost 500 by me).

Then there’s the actual Ravenports repo that mostly contains the buildsheets which are compiled from Ravensource. It has over 1.400 commits right now.

Installing the xfce-single-core meta-package

Finally there’s the repo for the Ravenadm command-line tool. It’s approaching 900 commits since February 2017.

There’s still more to Raven like the Pkg package manager from FreeBSD (that was modified to add Zstd compression support) or libbsd4sol, a portability library which allows building code on Solaris that uses BSDisms (which was needed to add support for that platform to Raven). Most of the work on all repos was done by John alone.

With over 100 pull requests and more than 20 issues it’s clear now that there’s some interest in the project. Raven is still very small, though, with 6 people haveing contributed ports so far. After learning the basics and opening pull requests for half a year, I’ve been granted write-access to the source repository. Just recently I was able to push my 100th active port (there have been ports that became obsolete and were removed).

In general I’d say that there could of course be more people around and that the project would benefit from being able to provide more packages – though more than 3.200 is not bad at all! Also it’s good that there seems to be a growing user base which is even more important than having more porters join in. From my point of view, Raven is a healthy and fast-moving project. Still young, but doing well and heading in the right direction.

Major changes

There have been some pretty big changes that happened with Raven over time. Initially John started with a GCC6-based toolchain, only to switch to GCC7 when that was released. That was before my time with the project, but I witnessed the switch to GCC8.

Changing the toolchain certainly is a major interruption and most people are advised to just wait for the official repository to be re-rolled and then update. I had some bad luck in this regard – literally the day after I finally completed a working (and almost complete) set of basic packages for the FreeBSD_i386 platform, I faced the change to GCC8. Due to a lack of time I still haven’t repeated the switch on i386 (but I still plan to do it sometime).

The thunar file manager

Other changes that always have a huge impact (causing lots and lots of packages to be rebuilt) is adopting a new version (as well as dropping an old one) of the popular interpreter languages like Python, Perl and Ruby. Ravenports always supports two versions of Perl and Ruby and two versions of Python 3 (as well as 2.7 for now). So when Python 3.7 was released, 3.5 was removed and Perl 5.24 had to go when 5.28 was added.

Recently the former LLVM port that included everything regarding LLVM was split (LLVM, Clang, lld, openmp). Also now and then new statements are added to Ravenadm, so that old versions cannot work with a new release of the buildsheet repository (which is called “conspiracy”). But this is pretty easy to work around compared to the changes mentioned before.

So on the whole, Raven has proven that it can easily stand even big changes. For me this is essential to build faith in a project. And Raven is doing well in this regard.

Desktop-ready?

There are lots of people who will want to use Raven on servers. That’s totally fine of course. But for a project as ambitious as Ravenports, it’s necessary to provide a somewhat comfortable environment for the developers and the users alike. If it doesn’t manage to become a daily driver for people it cannot succeed.

For that reason I decided to work towards good desktop support for the little dev machine that I dedicated to my work on the project. When I started, X11 was already working and Openbox had freshly landed in the repos. So I had a simplistic environment to work with: Openbox + Xterm. However I could not even change my keyboard layout! Therefor I wrote a port for setxkbmap and eventually it was accepted as the first outside contribution to the project.

The Surf web browser

Next I did some work to get the FLTK toolkit and the EDE desktop in. Then I added my favorite terminal emulator, Sakura. This worked out pretty well and the biggest shortcoming at the end of 2017 was that there was no real graphical browser available. A lot has changed since then!

Desktop choices

Today you can choose between multiple window managers, both floating and tiling:

• twm
• cwm
• openbox
• fluxbox
• xfwm4
• pekwm
• i3

And in case you prefer a real desktop environment, there are also several available:

• Lumina (moderate, Qt-based)
• Xfce4 (somewhat light-weight, GTK-based)
• EDE (extremely frugal and minimalistic, FLTK-based)

Two graphical web browsers are available, Surf (which is deliberately simplistic and does not even support tabs) as well as an old version of Firefox (the last one that builds without Rust). This is certainly not perfect but much better than a year before.

Also other important programs are available, including LibreOffice! Last month the Apache webserver landed – which is a pretty complex port compared to many others.

Shortcomings

Are there packages you’ll miss? Most certainly. However there’s a wishlist now with ports that people would like to see created (please feel free to add more requests there). And that’s another good step ahead. Currently it’s almost 120 items long. Fortunately there’s been some success, too, and 26 requested ports have been created and taken of the list so far.

There are some future ports that will require lots of effort (hint: Help wanted!). The most important one that blocks some other important ports is the Rust compiler. There has been some work done on this but it’s not done, yet. Another real beast is TeX. This totally must be supported at some point. Current versions of Firefox and Chromium are often asked for. And somebody even requested Eclipse (which needs Java!). So there’s definitely more than enough work to do.

Using Raven on Linux works, but there are some flaws. Initially the Pkg package manager used to crash quite often. John traced that back to a bug in the version of SQlite that’s used internally by Pkg: The problem only struck on Linux and was fixed by using a newer version instead. While it’s much better now, there’s still the occasional problem with it.

While the packages from the repo work finde on Solaris 10u8 and above as well als Illumos, the exact version 10u8 is currently required to build packages. This is due to Solaris not being able to work with older system libraries in the build chroot. It would be great to haven an alternative ravensys-root for any Illumos distribution (OmniOS, SmartOS, Tribblix, …) available so that interested people without access to that specific closed-source Solaris version can develop Raven on that platform.

I don’t know how well Raven works on Darwin. Since I don’t have access to any macOS machines and PureDarwin is not really ready, yet, there’s currently no chance for me to test it. I intend to buy an older MacBook or something in the future, though, if I come across a fair offer and have some money available to spend on my hobby.

Some ports are not available on one platform or the other: Illumos mostly because they’d require patches to build and Linux often because it relies on additional libraries that have not yet been added to Raven. And then there’s a lot of packages that are mostly untested. All of these issues can be fixed, of course. All of those require a larger user-base, though. So it’s probably the best strategy to keep working on making Raven attractive to more users and address things when the right people show up.

What’s to come?

Currently Raven uses the primordial X11 input drivers (xf86-input-keyboard and xf86-input-mouse) on all platforms. In 2013 Linux pioneered support for generic input drivers by exposing the kernels “event devices”. Not too much later many Linux distributions adopted xf86-input-evdev. In 2014 there was a GSOC project to add evdev support for FreeBSD. Like many projects it came along a good part of the way but eventually was left unfinished. It was picked up and completed by a FreeBSD developer in 2016.

Xfce’s settings and applications menu

To use it, a special kernel had to be built so it would expose /dev/input device nodes. Then a sysctl had to be set – and eventually X11 had to be patched for emulated udev support… Why would anybody want to do all this just for different input drivers? Multi-touch support is just one valid reason. Another one is that having evdev-based input drivers is half the way to eventually support libinput, too. And that is one of the prerequisites for Wayland!

This month FreeBSD has finally enabled evdev support in the GENERIC kernel in both -CURRENT and 12-STABLE. That means the upcoming FreeBSD 12.0 will not support it out of the box, but most likely a future 12.1 will. Dragonfly BSD has also grown support for event devices and people are interested in working towards Wayland. I hope that we’ll be able to get xf86-input-evdev working with our X11 (on Dragonfly, FreeBSD and Linux) next year,

I’m taking a little break from Xfce now (but plan to port most of the remaining components later to make it a well-supported DE in Raven). There are a few things I have planned like adding Linux support for OpenVPN (it depends on some libraries and programs that are Linux only which are not yet in Raven). Also I intend to take a look at adding some more Qt5 components and write a few requested ports. And finally I want to write another post next year – a tutorial on using Ravenports and creating new ports.

So keep flying with us – it’s exciting times!

Advertisements

Exploring OmniOS in a VM (2/2)

This is the second part of my post about “exploring OmniOS in a VM”. The first post showed my adventures with service and user management on a fresh installation. My initial goal was to make the system let me ssh into it: Now the SSH daemon is listening and I’ve created an unprivileged user. So the only thing still missing is bringing up the network to connect to the system from outside.

Network interfaces

Networking can be complicated, but I have rather modest requirements here (make use of DHCP) – so that should not be too much of a problem, right? The basics are pretty much the same on all Unices that I’ve come across so far (even if Linux is moving away from ifconfig with their strange ip utility). I was curious to see how the Solaris-derived systems call the NIC – but it probably couldn’t be any worse than enp2s0 or something that is common with Linux these days…

# ifconfig
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
          inet 127.0.0.1 netmask ff000000
lo0: flags=200200849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
          inet6 ::1/128

Huh? Only two entries for lo0 (IPv4 and v6)? Strange! First thought: Could it be that the type of NIC emulated by VirtualBox is not supported? I looked for that info on the net – and the opposite is true: The default VirtualBox NIC (Intel PRO/1000) is supported, the older models that VBox offers aren’t!

Ifconfig output and fragment of the corresponding man page

Obviously it’s time again to dig into manpages. Fortunately there’s a long SEE ALSO section again with ifconfig(1M). And I’ve already learned something so far: *adm commands are a good candidate to be read first. Cfgadm doesn’t seem to be what I’m looking for, but dladm looks promising – the short description reads: “Administer data links”.

I must say that I like the way the parameters are named. No cryptic and hard to remember stuff here that makes you wonder what it actually means. Parameters like “show-phys” and “show-link” immediately give you an idea of what they do. And I’m pretty sure: With a little bit of practice it will work out well to guess parameters that you’ve not come across, yet. Nice!

# dladm show-link
LINK         CLASS     MTU    STATE     BRIDGE     OVER
e1000g0      phys      1500   unknown   --         --

Ok, there we have the interface name: e1000g0. I don’t want to create bridges, bond together NICs or anything, so I’m done with this tool. But what to do with that information?

The ifconfig manpage mentions the command ipadm – but it’s kind of hidden in the long text. For whatever reason it’s missing in SEE ALSO! I’d definitely suggest that it’d be added, too. More often than not people don’t have the time to really read through a long manpage (or are not as curious about a new system as I am and thus don’t feel like reading more than they have to). But anyways.

Ipadm manpage

# ipadm create-if e1000g0

This should attach the driver and create the actual interface. And yes, now ifconfig can show it:

Interface e1000g0 created

Network connection?

Almost there! Now the interface needs an IP address. Looks like ipadm is the right tool for this job as well. I had some trouble to find out what an interface-id is, though. The manpage obviously assumes the reader is familiar with that term, which was not the case with me. I tried to find out more about it, but was unable to locate any useful info in other manpages. So I resorted to the OmniOS wiki again and that helped (it seems that you can actually choose almost anything as an Interface ID but there are certain conventions). Ok, let’s try it out and see if it works:

# ipadm create-addr -T dhcp e1000g0/v4

No error or anything, so now the system should have acquired an IPv4 address.

IPv4 address assigned via DHCP

10.0.2.15, great! Let’s see if we can reach the internet:

# ping elderlinux.org
ping: unknown host elderlinux.org

DNS pt. 1

Ok, looks like we don’t have name resolution, yet. Is the right nameserver configured?

# cat /etc/resolv.conf
cat: cannot open /etc/resolv.conf: No such file or directory

Oops. There’s something not right here! If I configure my FreeBSD box to use DHCP that makes sure resolv.conf is populated properly. The interface e1000g0 got an IP – so the DHCP request must have been successful. But did something break before the network was configured completely? Is the DHCP client daemon even running?

DHCP

# ps aux | grep -i [d]hcp
root       551  0.0  0.1 2836 1640 ?        S 18:52:22  0:00 /sbin/dhcpagent

Hm! I’m used to dhclient, but dhcpagent is unknown to me. According to the manpage, it’s the DHCP client daemon, so that must be what OmniOS uses to initiate and renew DHCP requests. And obviously it’s running. However the manpage solves the mystery:

Aside from the IP address, and for IPv4 alone, the netmask, broadcast address, and the default router, the agent does not directly configure the workstation, but instead acts as a database which may be interrogated by other programs, and in particular by dhcpinfo(1).

Ah-ha! So I have to configure things like DNS myself. Nevertheless the system should have received the correct nameserver IP. Let’s see if we can actually get it via the dhcpinfo command that I’ve just learned about. A look at the manpage as well as at the DHCP inittab later I know how to ask for that information:

# dhcpinfo -i e1000g0 DNSserv
192.168.2.1

Right, that’s my nameserver.

Some lines from /etc/dhcp/inittab

Route?

Is the routing information correct? Let’s check.

# netstat -r -f inet
[...]
default      10.0.2.2  [...]
10.0.2.0     10.0.2.15 [...]
localhost    localhost [...]

Looks good and things should work. One more test:

# route get 192.168.2.1
   route to: fw.local
destination: default
       mask: default
    gateway: 10.0.2.2
  interface: e1000g0
[...]

Alright, then it’s probably not a network issue… But what could it be?

DNS pt. 2

Eventually I found another hint at the wiki. Looks like by default OmniOS has a very… well, old-school setting when it comes to sources for name resolution: Only the /etc/hosts is used by default! I haven’t messed with nsswitch.conf for quite a while, but in this case it’s the solution to this little mystery.

# fgrep "hosts:" /etc/nsswitch.conf
hosts:      files

There are a couple of example configurations that can be used, though:

# ls -1 /etc/nsswitch.*
/etc/nsswitch.ad
/etc/nsswitch.conf
/etc/nsswitch.dns
/etc/nsswitch.files
/etc/nsswitch.ldap
/etc/nsswitch.nis

Copying nsswitch.dns over nsswitch.conf should fix that problem.

And really: Instead of getting a ping: unknown host elderlinux.org now I get no answer from elderlinux.org – which is OK, since my nameserver doesn’t answer ping requests.

Remote access

Now with the network set up correctly, it’s finally time to connect to the VM remotely. To be able to do so, I configure VirtualBox to forward port 22 of the VM to port 10022 on the host machine. And then it’s time to try to connect – and yes, it works!

SSHing into the OmniOS VM… finally!

Conclusion

So much for my first adventure with OmniOS. I had quite a few difficulties to overcome even in this very simple scenario of just SSHing into the VM. But what could have been a trivial task proved to be rather educational. And being a curious person, I actually enjoyed it.

I’ll take a break from the Illumos universe for now, but I definitely plan to visit OmniOS again. Then I’ll do an installation on real hardware and plan to take a look at package management and other areas. Hope you enjoyed reading these articles, too!

Exploring OmniOS in a VM (1/2)

While I’ve been using Unix-like systems for quite a while and heavily for about a decade, I’m completely new to operating systems of the Solaris/Illumos family. So this post might be of interest for other *BSD or Linux users who want to take a peek at an Illumos distribution – or to the Illumos people interested in how accessible their system is for users coming from other *nix operating systems.

In my previous post I wrote about why I wanted to take a look at OmniOS in the first place. I also detailed the installation process. This post is about the first steps that I took on my newly installed OmniOS system. There’s a lot to discover – more actually than I first thought. For that reason I decided to split what was meant to be one post into two. The first part covers service management and creating a user.

Virtualize or not?

According to a comment, a Reddit user would be more interested in an installation on real hardware. There are two reasons why I like to try things out using a hypervisor: 1) It’s a quick thing to do and no spare hardware is required 2) it’s pretty easy to create screenshots for an article like this.

OmniOS booted up and root logged in

However I see the point in trying things out on real hardware and I’m always open to suggestions. For that reason I’ll be writing a third post about OmniOS after this one – and will install it on real hardware. I’ve already set a somewhat modern system aside so that I can test if things work with UEFI and so on.

Fresh system: Where to go?

In the default installation I can simply login as root with a blank password. So here we are on a new OmniOS system. What shall we explore first?

The installer asked me which keymap to use and I chose German. While I’m familiar with both the DE and US keymaps enough that I can work with them, I’ve also learned an ergonomic variant called Neo² (think dvorak or colemak on steroids) and strongly prefer that.

It’s supported by X11 and after a simple setxkbmap de neo -option everything is fine. On the console however… For FreeBSD there’s a keymap available, but for Illumos-based systems it seems I’m out of luck.

So here’s the plan: Configure the system to let me SSH into it! Then I can use whatever I want on my client PC. Also this scenario touches upon a nice set of areas: System services, users, network… That should make a nice start.

Manpages (pt. 1)

During the first startup, smf(5) was mentioned so it might be a good idea to look that up. So that’s the service management facility. But hey, what’s that? The manpage clearly does not describe any type of configuration file. And actually the category headline is “Standards, Environments, and Macros”! What’s happening here?

smf(5) manpage

First discovery: The manpage sections are different. Way different, actually! Sections like man1, man1b, man1c, man3, man3bsm, man3contract, man3pam, etc… Just take a look. Very unfamiliar but obviously clearly arranged.

The smf manpage is also pretty informative and comprehensive including the valuable see also section. The same is true for other pages that I’ve looked at so far. On the whole this left a good impression on me.

System services

Solaris has replaced the traditional init system with smf and Illumos inherited it. It does more than the old init did, though: Services are now supervised, too. If a service is meant to be kept running, smf can take care of restarting it, should it die. It could be compared to systemd in some regards, but smf was created earlier (and doesn’t have the same … controversial reputation).

Services are described by a Fault Management Resource Identifier or FMRI which looks like svn:/network/loopback:default but can be shortened if they are unambiguous. I had no idea how to work with all this but the first “see also” reference was already helpful: scvs can be used to view service states (and thus get an idea about what the system is doing anyway).

svcs: service status

Another command caught my attention right away, too: svcadm sounded like it might be useful. And indeed this was just what I was searching for! The manpage revealed a really straight-forward syntax, so here we go:

# svcadm enable sshd
svcadm: Pattern 'sshd' doesn't match any instances
# svcadm enable ssh

The latter command did not produce any output. And since we’re in Unix territory here, that’s just what you want: It’s the normal case that something worked as intended, no need for output. Issuing a svcs and grepping for ssh shows it now so it should be running. But is SSH really ready now? Let’s check:

# sockstat -4 -l
-bash: sockstat: commmand not found

Yes, right, this is not FreeBSD. One netstat -tulpen later I know that it’s not exactly like Linux, either. Once more: man to the rescue!

# netstat -af inet -P tcp

The output (see image below) looks good. Let’s do one last check and simply connect to the default SSH port on the VM: Success. SSHd is definitely running and accepting connections.

Testing if the SSH daemon is running

Alright, so much for very basic service management! It’s just a test box, but I don’t really like sshing into it as root. Time to add a user to the system. How hard could that be?

User management

Ok, there’s no user database like on FreeBSD or anything like that. It’s just plain old /etc/passwd and /etc/shadow. How boring. So let’s just create a user and go on with the next topic, right?

# useradd -m kraileth
UX: useradd: ERROR: Unable to create home directory: Operation not applicable.

Uh oh… What is this? Maybe it’s not going to be so simple after all. And really: Reading through the manpage for useradd, I get a feeling that this topic is everything – but certainly not boring!

There’s a third file, /etc/user_attr, involved in user management. Users and roles can be assigned extended attributes there. Users can have authorizations, roles, profiles and be part of a project. I won’t go into any detail here (it makes sense to actually read the fine manpages yourself if you’re interested). Now if you’re thinking that some of this might be related to what FreeBSD does with login.conf, you’re on the right track. I cannot claim that I understood everything after reading about it just once. But it is sufficient to get an idea of what sort of complex and fine-grained user management Illumos can do!

Content of /etc/user_attr

Manpages (pt. 2)

Ok, while this has been an interesting read and is certainly good to know, it didn’t solve the problem that I had. The useradd manpage even has a section called DIAGNOSTICS where several possible errors are explained – however the one that I’m having isn’t. And that’s a pitty, since some of the ones listed here are pretty self-explanatory while “Operation not applicable” isn’t (at least for me).

I read a bit more but didn’t find any clue to what’s going on here. When my man skills failed me I turned to documentation on the net. And what better place to start with than the OmniOSce Wiki?

When it comes to local users, the first sentence (ignoring the missing word) reads: On OmniOS, home is under automounter control, so the directory is not writable.

Ah-ha! That sounds quite a bit like the reason for the mysterious “not applicable”! That should be hint enough so I can go on with manpages, right?

# apropos automounter
/usr/share/man/whatis: No such file or directory
# man -k automounter
/usr/share/man/whatis: No such file or directory

Hm… Looks like the whatis database does not exist! Let’s create it and try again:

# man -w
# apropos automounter
# apropos automount
autofs(4)      - automount configuration properties
automount(1m)  - install automatic mount points
automountd(1m) - autofs mount/unmount daemon

There we go, more pages to read and understand.

User’s home directories

The automounter is another slightly complex topic (and the fact that the wiki mentions /export/home while useradd seems to default to /home doesn’t help to prevent confusion, either). So I’m going to sum up what I found out so far:

It seems that people at Sun had the idea that it would be nice to be able work not only on your own workstation but at any other one, too. Managing users locally on each machine would be a nightmare (with people coming and going). Therefore they created the Yellow Pages, later renamed to NIS (Network Information Service). If you have never heard of it, think LDAP (as that has more or less completely replaced NIS today). Thus it was possible to get user information over the net instead of from local passwd and friends.

The logical next step was shared home directories so employees could have fully networked user logins on multiple machines. Sun already had NFS (Network File System) which could be used for the job. But it made sense to accompany it with the automounter. So this is the very short story of why home directories are typically located in /export/home on Solaris-derived operating systems: They were meant to be shared via NFS!

So we have to add a line to /etc/auto_home to let the automounter know to handle our new home directory:

* localhost:/export/home/&

Configuring the automounter for the home directory

Most of the two automounter configuration files are made up of the CDDL (pronounced “cuddle”) license – I’ve left it out here by using tail (see picture). After adding the needed rule (following the /export/home standard even though I don’t plan on using shared home directories), the autofs daemon needs to be restarted, then the user can finally be created:

# mkdir /export/home
# useradd -m -b /export/home kraileth
# passwd kraileth

Creating a new user

So with the user present on the system, we should be able to SSH into the local machine with that user:

ssh kraileth@localhost

SSHing into the system locally

Success! Now all that remains is bringing the net up.

What’s next?

The next post will be mostly network related and feature a conclusion of my first impressions. I hope to finish it next week.

A look beyond the BSD teacup: OmniOS installation

Five years ago I wrote a post about taking a look beyond the Linux teacup. I was an Arch Linux user back then and since there were projects like ArchBSD (called PacBSD today) and Arch Hurd, I decided to take a look at and write about them.

Things have changed. Today I’m a happy FreeBSD user, but it’s time again to take a look beyond the teacup of operating systems that I’m familiar with.

Why Illumos / OmniOS?

There are a couple of reasons. The Solaris derivatives are the other big community in the *nix family besides Linux and the BSDs and we hadn’t met so far. Working with ZFS on FreeBSD, I now and then I read messages that contain a reference to Illumos which certainly helps to keep up the awareness. Of course there has also been a bit of curiosity – what might the OS be like that grew ZFS?

Also the Ravenports project that I participate in planned to support Solaris/Illumos right from the beginning. I wanted to at least be somewhat “prepared” when support for that platform would finally land. So I did a little research on the various derivatives available and settled on the one that I had heard a talk about at last year’s conference of the German Unix Users Group: “OmniOS – Solaris for the Rest of Us”. I would have chosen SmartOS as I admire what Bryan Cantrill does but for getting to know Illumos I prefer a traditional installation over a run-from-RAM system.

There was also a meme about FreeBSD that got me thinking:

Internet Meme: Making fun of FreeBSD

Of course FreeBSD is not run by corporations, especially when compared to the state of Linux. And when it comes to sponsoring, OpenBSD also takes the money… When it comes to FreeBSD developers, there’s probably some truth to the claim that some of them are using macOS as their desktop systems while OpenBSD devs are more likely to develop on their OS of choice. But then there’s the statement that “every innovation in the past decade comes from Solaris”. Bhyve alone proves this wrong. But let’s be honest: Two of the major technologies that make FreeBSD a great platform today – ZFS and DTrace – actually do come from Solaris. PAM originates there and a more modern way of managing services as well. Also you hear good things about their zones and a lot of small utilities in general.

In the end it was a lack of time that made me cheat and go down the easiest road: Create a Vagrantfile and just pull a VM image of the net that someone else had prepared… This worked to just make sure that the Raven packages work on OmniOS. I was determined to return, though – someday. You know how things go: “someday” is a pretty common alias for “probably never, actually.”

But then I heard about a forum post on the BSDNow! podcast. The title “Initial OmniOS impressions by a BSD user” caught my attention. I read that it was written by somebody who had used FreeBSD for years but loathed the new Code of Conduct enough to leave. I also oppose the Conduct and have made that pretty clear in my February post [ ! -z ${COC} ] && exit 1. As stated there, I have stayed with my favorite OS and continue to advocate it. I decided to stop reading the post and try things out on my own instead. Now I’ve finally found the time to do so.

First attempt at installing the OS

OmniOS offers images for three branches: Stable, LTS and Bloody. Stable releases are made available twice a year with every fourth release being supported for three years (LTS) instead of one. Bloody images are more or less development snapshots meant for advanced users who want to test the newest features.

I downloaded the latest stable ISO and spun up a VM in Virtual Box. This is how things went:

Familiar Boot Loader

Ah, the good old beastie menu – with some nice ASCII artwork! OmniOS used GRUB before but not too long ago, the FreeBSD loader was ported over to Illumos. A good choice!

Two installers available

It looks like the team has created a new installer. I’m a curious person and want to know what it was like before – so I went with the old text-based installer.

Text installer: Keymap selection

Not much of a surprise: The first thing to do is selecting the right keymap.

ZFS pool creation options

Ok, next it’s time to create the ZFS pool for the operating system to install on. It seems like the Illumos term is rpool (resource pool I guess?). Since I’m just exploring the OS for the first time, I picked option 1 and… Nothing happened! Well, that’s not exactly true, since a message appears for a fraction of a second. If I press 1 again, it blinks up briefly again. Hm!

I kept the key pressed and try my best to read what it’s saying:
/kayak/installer/kayak-menu[254]: /kayak/installer/find-and-install: not found [No such file or directory]

Oops! Looks like there’s something broken on the current install media… So this was a dead-end pretty early on. However since we’re all friends in Open Source, I filed an issue with OmniOS’s kayak installer. A developer responded the next day and the issue was solved. This left a very good impression on me. Quality in development doesn’t show in that you never introduce bugs (which is nearly impossible even for really lame programs) but in how you react to bugs being found. Two thumbs up for OmniOS here (my latest PRs with FreeBSD have been rotting for about a year now)!

Dialog-based installer

What a great opportunity to test the new installer as well! Will it work?

Dialog-based installer: Keymap selection

Back on track with the dialog-based installer. Keymap selection is now done via a simple menu.

ZFS pool creation options

Ok, here we are again! Pool creation time. In the new installer it just does its job…

Disk selection

… and finds the drives, giving me a choice on where to install to. Of course it’s a pretty easy decision to make in case of my VM with just one virtual drive!

ZFS Root Pool Configuration

Next the installer allows for setting a few options regarding the pool. It’s nice to see that UEFI seems to be already supported. For this VM I went with BIOS GPT, though.

Hostname selection

Then the hostname is set. For the impatient (and uncreative) it suggests omniosce. There’s not too much any installer could do to set itself apart (and no need to).

Time zone selection 1

Another important system configuration is time zone settings. Since there’s a lot of time zones, it makes sense to group them together by continent instead of providing one large list. This looks pretty familiar from other OS installations.

Time zone selection 2

The next menu allows for selecting the actual time zone.

Time zone confirmation

Ok, a confirmation screen. A chance to review your settings probably doesn’t hurt.

Actual copying of OS data

Alright! Now the actual installation of the files to the pool starts.

Installer: Success!

Just a moment later the installation is finished. Cool, it even created a boot environment on its own! Good to see that they are so tightly integrated into the system.

Last step

Finally it’s time to reboot. The installer is offering to do some basic configuration of the system in case you want to do that.

Basic configuration options

I decided not to do it as you probably learn most when you force yourself to figure out how to configure stuff yourself. Of course I was curious, though, and took a peek at it. If you choose to create a user (just did this in another VM, so I can actually write about the installer), you’ll get to decide if you want to make it an administrative user, whether to give it sudo privileges and if you want to allow passwordless sudo. Nice!

First start: Preparing services

After rebooting the string “Loading unix…” made me smile and I was very curious about what’s to come. On the first boot it takes a bit longer since the service descriptions need to be parsed once. It’s not a terribly long delay, though.

First login on the new system

And there we have it, my first login into an OmniOS system installed myself.

What’s next?

That’s it for part one. In part two I’ll try to make the system useful. So far I have run into a problem that I haven’t been able to solve. But I have some time now to figure things out for the next post. Let’s see if I manage to get it working or if I have to report failure!

Ravenports: Status update and the Dragonfly case

This is part three of a series of posts on cross-platform package management. The previous posts contained general thoughts about software packaging today and a somewhat in-depth overview on the Ravenports package system.

In this post I want to give some more background on why Ravenports might be interesting to some people and explain the Dragonfly case. I’ll also give you a little status update.

Real-world use case: Package homogenization

As mentioned in the previous post, Ravenports currently supports DragonflyBSD, FreeBSD-amd64, Linux-glibc-x86_64 and Solaris/Illumos-amd64.

There are of course minor differences between the platforms: The shadow package is only available on Linux because the other systems use different ways for scrambling user passwords, there’s no fuse for Dragonfly because that is not supported there and iocage is FreeBSD-only because it’s a jail manager. Also some packages don’t build on the SunOS platform, yet, because they need additional patches. When I wrote my previous post, e.g. the xorg-server package was not available for SunOS. It is now, though it’s not of too much use, because the Xorg drivers aren’t. Still you can see things are moving in the right direction.

Other than those special cases, Raven is consistent across all supported platforms (which is one of the major features that caught my attention after all): You can get packages of the same version on Linux and FreeBSD (and the other platforms) and – as much as feasible – they are also configured alike. It’s not guaranteed that the official repositories hold the same package versions at all times. If you require that, you currently have to roll your own repos but that’s not hard to do at all (I’ll write about actually using Raven in detail in a future post).

As I’m working in a heterogeneous environment, it is my hope that Raven will take a lot of the headaches away that result from native package management in the Linux jungle (Ubuntu has one version, Debian ships another, CentOS is stuck with an ancient one and patches is to use a different directory structure, … you get the point) and as a bonus offers the same version on *BSD. I’m pretty sure that I’m not alone in this jungle and while most of us are able to survive there, few would deny that it does require a big machete and some teeth grinding from time to time. It’s ok – but can’t we use our time for better things? I think so.

The Dragonfly case

At home I’m more or less exclusively a FreeBSD user and that’s the platform that I do my porting work on. I have in fact never seriously run Dragonfly – not actually because of a lack of interest in it but rather due to a very limited set of hardware available to me. I’ve never found a spare piece of hardware among my pool that would run the OS well. Still I’m dedicating most of this post to DragonflyBSD. Why? Because Raven is in a special position there.

As you may know, DragonflyBSD is by far the smallest of the four “big” BSD projects (some actually talk about only three and leave out Dfly). They are doing an amazing job for the little manpower that they have, but all small projects struggle to keep up with changes going on in the open-source ecosystem. Run your desktop on a *nix machine? Simply ask your package manager how many packages you have installed to make that possible and get a rough idea on just how much work needs to be put into maintaining all of them.

Most OSes and distributions maintain their own source repositories. Dragonfly never had the manpower to even consider this (also they have way different focus than just doing again what everybody else does). Historically this is why they used Pkgsrc as this portable package system looked like a great option for Dragonfly. A lot of work was put into it, but there were quite some issues with it.

Pkgsrc on Dragonfly

Some were of technical nature: Pkgsrc can do binary packages, but if you know other package managers, the old tools really pale in comparison. There were conflicts in the release model, as Pkgsrc’s quarterly releases were not well fit for Dfly. But most importantly: A lot of packages were really outdated and where updates did occur so did breakage.

Updates were only tested against NetBSD and so it happened quite often that a single update broke anything from a couple to thousands (!) of ports for DragonflyBSD. Not even in the latter case would Pkgsrc suggest to revert the update that caused so much trouble – Dragonfly was expected to fix all of the fallout themselves. To be fair, there surely was no intention to break anything. But there simply weren’t any test farms either, so even if porters would have liked to care better for Dfly, it wouldn’t have been easy for them.

For technical problems there’s usually a solution, especially if people are involved who both are knowledgeable on the topic and show great dedication. What’s difficult to solve however, are political problems. And that’s what arose in the relationship between Dragonfly and Pkgsrc: DragonflyBSD has officially been a first-class citizen and thus on-par with NetBSD. But because of the frequent breakage, Dragonfly users felt differently about it. Unfortunately, multiple attempts and suggestions made to improve the situation also led to nothing.

In the end it became clear that Pkgsrc primarily is NetBSD’s package system. Since one of NetBSD’s primary goals is portability, Pkgsrc is of course also portable. This portability comes at a price, though – and that price is the need for a whole army of Pkgsrc maintainers to support an OS besides NetBSD well. As Pkgsrc was chosen due to the lack of manpower, this fact learned the hard way, showed that it wasn’t the right solution.

Dports

Searching for a different one, John Marino stepped forward in creating an alternative: Dports.

In a nutshell this meant bringing FreeBSD’s new package manager (pkg a.k.a. “pkg-ng”) to Dragonfly as well as FreeBSD’s ports – with the changes necessary to make those build on this platform. It was a huge task but the advantages over the old system were big enough for the project to make sense. Eventually DragonflyBSD ditched Pkgsrc after more than half a decade and adopted Dports. If you want to know more, you can e.g. read the old comments here.

Of course Dports is not a “do once” effort; it constantly requires work to keep all the ports in sync with newer versions in FreeBSD’s ports collection where the active development takes place. And John didn’t have enough, he continued to experiment with packaging, writing Synth. He didn’t like some of the aspects of FreeBSD’s ports collection and on the other hand wanted features that were unlikely to find their way into the FPC.

Then FreeBSD introduced flavored ports. According to John, Dports had in general been less work for more packages available in comparison to Pkgsrc. However breakage still is an issue. And since the flavor-related changes in FreeBSD this has become much worse. So over time things have become more pressing for a real and permanent solution.

Ravenports to completely replace Dports?

John had been thinking about new ways of package creation for a long time. With Synth he now had his own package building system and with package numbers increasing far beyond the point of somewhat acceptable maintainance requirements, he decided to give a new project a try after all and this evolved into what is Ravenports today.

Currently John Marino is working on Ravenports while still keeping Dports up to date. Last month he announced that he’d like to step down from working on Dports because he considers his time better spend on Raven. The big question is now: Will somebody volunteer to claim maintainership for Dports? So far it doesn’t look like it. Which means: After more than 5 years in existence, Dports might actually go away. Ravenports might replace it as the official packaging system on DragonflyBSD.

For that reason John has asked the community what packages are important for people. Again, I’m not a Dragonfly user, but I’ve been very surprised by the response of the community – or rather the lack of it. Only a few people have responded so far and this makes me wonder if the majority of Dragonfly users have either missed this totally or didn’t realize what it means.

I’m convinced that Ravenports definitely is the superior system. However a transition would have a huge impact. When Dragonfly switched from Pkgsrc to Dports it meant that a whole lot more package were available. In case of a possible switch to Ravenports the opposite would happen: There’s roughly 30,500 packages in Dports currently while at the moment the Dragonfly repository of Ravenports holds exactly 3,600 packages!

Yes, the numbers cannot be compared 1:1, but it’s not too hard to see that it would mean a dramatic decrease in packages being available. On the plus side, software versions in Ravenports are often much newer than the same program available on FreeBSD. More and more packages will be added to Raven – but this takes time. For that reason it’s very important that you tell us what software you depend on so that it can be added with higher priority.

What’s being done

Some packages have already been identified to be a problem. An important one is Firefox. Ravenports has it available in the latest ESR version that doesn’t require Rust. But web browsers don’t age very well and newer versions of Firefox are practically a must-have. Rust cannot be easily added, however – it’s another problematic package. A third example is everything TeX which is a beast of a project.

One user requested more components of the Xfce desktop. I had started porting Xfce for my previous article anyway and I’m planning to either port the complete desktop or at least the most important parts. Next is Thunar, Xfce’s file manager. But to give you an idea of what this means: Before I can even think about porting it, I need to get in a whole lot of dependencies first. A quick look at it showed that I might have to create up to 30 ports for that (including some that probably won’t be trivial). So that’s not work for one or two weeks but instead will likely take a lot longer.

Also I’m doing some work on getting FreeBSD-i386 working. It’s in fact mostly done; I have some changed ports that I need to commit and one last package doesn’t work to publish an i386 repo that is self-hosting. However this is more or less a thing that I’m doing for fun and to learn.

If I succeed with that, I plan on backporting Raven to FreeBSD 10 (amd64) and then 9. The reason for that is that I hope to make it run on MidnightBSD. It is a fork of FreeBSD and technically close to 9. As another (even smaller) BSD it also has the known problems in keeping up with all current package versions in their mports. So it could make sense to join efforts. But this is just an idea, I haven’t approached Lucas Holt, yet, and I won’t before I have something to show off.

Another platform that will likely be supported, is MacOS, further backing Ravenport’s promise of being a universal package system. And Linux support will be improved in the future as well. Currently distributions that ship very old versions of glibc (e.g. RedHat) cannot use Raven’s packages. Possible solutions to that are being discussed.

How YOU can help

Are you running a *nix operating system supported by Ravenports? In that case you can help. Do the bootstrap and start using it – it’ll install to /raven by default so there should be no conflicts with your other package system. If you’re using it, please provide feedback and create issues if you find problems! The more people actually use the packages, the more confident we can all be that those work well on all platforms.

If you have a little more interest in packaging systems, you can try to create a port and see if you like it. It does help if you’re familiar with FreeBSD’s ports, Dports or really any other package system, but that’s not a requirement. I started with practically no prior knowledge and now after less than a year, I’m maintaining over 80 ports. I’m not a programmer and don’t know much about Make and such. For me it has been a great learning experience.

Oh, and if I can do it, so can you. I intend to write another post on using Raven and at least one more about writing your own ports. If you want to give it a try before, feel free to contact me, I’ll try to help. Also any questions are appreciated so I get a better idea if what I should write about.

Ravenports: A modern, cross-platform package solution

This post is about Ravenports, a universal package system und building framework for *nix systems (DragonflyBSD, FreeBSD, Linux and Solaris at the time of this writing). It’s a relatively young project that begun in late February 2017 after a longer period of careful planning. The idea is to provide a unified, convenient experience in a cross-platform way while putting focus on performance, scalability and modern tooling.

What exactly is it and why should you care? If you’ve read my previous post, you know that I consider the old package systems lacking in several ways. For me Raven already does a great job at solving some problems existing with other systems – and it’s still far from tapping its full potential.

Rationale

A lot of people will think now: “We already have quite capable package systems. What’s the point in doing it again?” Yes, in many regards it’s “re-inventing the wheel”… And rightfully so! Most of the known package systems are pretty old now and while new features were of course added, this is sometimes problematic. There is a point where it’s an advantage to start fresh and incorporate modern ideas right from the start. Being able to benefit from the experience and knowledge gained by using the other systems for two decades when designing a new system is invaluable.

Ravenadm running on FreeBSD, OmniOS, Ubuntu Linux and DragonflyBSD

Ravenports was designed, implemented and is primarily maintained by a veteran in packaging software. John Marino at a time maintained literally thousands of ports for FreeBSD and DragonflyBSD. In addition to that, he wrote an alternative build tool called Synth. Aiming for higher portability, he modified Synth to work with Pkgsrc (which is available for many platforms) and also ported the modern Pkg package manager from FreeBSD to work with it.

In the end he had too many ideas about what could be improved in package building that would not fit into any existing project. Eventually Ravenports was born when he decided to give it a try and create a new framework with the powerful capabilities that he wanted to have and without the known weaknesses of the existing ones.

How does it compare to xyz?

It probably makes sense to get to know Ravenports by comparison to others. Let’s take a look at some of them first:

1) FreeBSD’s ports system is the oldest one such framework. It’s quite easy to use today, very flexible and since the introduction of Pkg (or “pkg-ng”) it also has a really nice package manager.
2) NetBSD adopted the ports system and developed it according to their own needs. It’s missing some of the newer features that FreeBSD added later but has gained support for an incredible amount of operating systems. Unfortunately it still uses the old pkg_* tools that really show their age now.
3) OpenBSD also adopted the early FreeBSD ports system. They took a different path and added other features. OpenBSD put the focus on avoiding users having to compile their own packages. To do so, they added so-called package flavors. This allows for building packages multiple times with different compile-time options set. Their package tools were re-written in Perl and do what they are meant to. But IMO they don’t compare well to a modern package manager.
4) Gentoo Linux with its portage system has taken flexibility to the extreme. It gives you fine-grained control over exactly how to build your software and really shines in that. The logical consequence is that, while it supports binary packages, this support is rudimentary in comparison.

EDE desktop, pekwm with Menda theme and brand-new LibreOffice

FreeBSD gained support for flavors in December 2017 and NetBSD did some work to support subpackages in a GSoC project in the same year. It’s hard to retrofit major new features into an existing framework, tough. When Ravenports started in the beginning of 2017, it already had those two features: Variant packages (Raven’s name for flavors) and subpackages. As a result they feel completely natural and fit well into the whole framework (which is why they are used excessively).

Ravenports knows ports options that can be set before building a package. Like with NetBSD or OpenBSD there’s generally fewer options available compared to FreeBSD. This is because Raven is more geared towards building binary packages than being a ports framework to build on the target machine (which would defeat the goal of always providing a clean building environment). For that reason the options mostly exist to support the variants for the packages. Compared to NetBSD’s Pkgsrc, Ravenports supports much fewer operating systems right now but has a much easier bootstrap process (binary!) for all supported platforms. It also offers a much superior package manager. When comparing against FreeBSD, OpenBSD and Gentoo, Ravenports is much more portable and supports multiple operating systems and – with the exception of FreeBSD – comes with a more modern package manager for binary packages.

Strong points

As Ravenports is not tied to a single operating system, it didn’t have to take into account specific needs that are for one OS only. In general there are no second-class citizens among the supported platforms. Also it was made to be agnostic of the package manager used. Right now it’s using Pkg only but other formats could be supported and thus binary packages be installed via pacman, rpm, dpkg, you-name-it.

Repology: Raven’s package freshness in percent (06/25/2018)

It allows for different versions of some software to be concurrently installed. If you e.g. want PHP 7.2 while some of your projects are stuck with 5.6 this is not a problem. It’s also possible to define a default version for databases like MySQL and Postgres as well as languages like Perl, Python and Ruby. Speaking of MySQL: Raven knows about Oracle MySQL, MariaDB, Percona and Galera. Only the first one is currently available (the ports for the others are missing) but the selection of which product to install is already present and the others can be easily added as needed.

If you build packages yourself you’ll notice that the whole tooling is fully integrated. Everything was planned right from the beginning to interact well and thus plays together just great. Also performance is something where Raven shines: Thanks to being programmed for high concurrency, operations like port scans are amazingly fast (if you know other frameworks).

Repology: Raven’s outdated package count (06/25/2018)

Raven follows a rolling-release model with extremely current package versions. In Repology, a fine tool for package maintainers and people interested in package statistics, Ravenports is the clear leader when it comes to freshness of the package repository: It rarely falls below 98% of freshness (while no other repo has managed to even reach 90% – and Repology lists almost 200 repositories!). If it does, it’s usually for less than a day until updates get pushed.

This is only possible because much of ports maintenance is properly automated. This saves a lot of work and allows for keeping the software version current without the need for dozens of maintainers. Custom port collections are supported if you have special needs like sticking to specific program versions. This way Raven can e.g. support legacy versions that should not be part of the main tree. It might also be interesting for companies that want to package their product for multiple platforms but need to keep the source closed. Ravenports supports private GitHub repositories for cases like this. All components of project itself are completely open-source, though, and are permissively licensed.

Also Raven is not the jealous kind of application. Packages are installed into /raven by default (you can choose to build your packages with a different prefix if you wish) and thus probably separate from the default system location for software. This makes it possible to use raven in addition to your operating system’s / distribution’s package manager instead of being forced to replace it.

Shortcomings

If you ask me about permanent problems with Raven: I don’t really see any. However there’s definitely a couple of things where it’s currently way behind other package systems. Considering how young the project is this is probably no wonder.

It’s a “needs more everything” situation. In fact it has the usual “chicken egg problem”: More available ports would be nice and potentially attract more users. With more users probably more people would become porters. And with more porters there’d surely be more ports available… But every new project faces problems like this and with resolve, dedication and perseverance as well as a fair amount of work, it’s possible to achieve the goal of making a project both useful and appealing enough for others to join in. Once that happens things get easier and easier.

KeePassXC, Geany and the EDE application menu

The Ravenports catalog has over 3,000 entries right now. It’s extremely hard to compare things like the package count, though. John provided an example: FreeBSD has 8 ports for each PostgreSQL version. With 5 supported versions that’s 40 ports. Ravenports has 5 ports with 8 subpackages each. In this case the package count is comparable, but not the port count. Taking flavors and multiversions into account, all repositories look much bigger than they actually are in case of available software. Also how to measure the quality of packages? What’s with ports that are used by less than a handful of people? What with those that are extremely outdated? Do you think they should count? It’s probably best to take a look and see if the software that you need is available. It is true though, that there’s of course still many important packages missing. IMO the most important one being Rust – which is not only needed for current versions of Firefox but increasingly important to build other software, too.

Also Linux support is not perfect, yet, and Solaris support even less so. On Solaris systems Raven is currently mostly binary-only because the Solaris kernel is unable to work with system libraries other than the ones matching exactly in version. Packages built on older releases of the OS work fine on newer ones, but for each OS release, a specific build environment would need to be created before building packages is possible. This is an issue that needs to be resolved in the future (I guess some help from the Illumos/Solaris community wouldn’t hurt). Also there are packages that don’t build on Solaris without patches which are not currently available. In case of important packages this leads to blockers since all other ports which depend on one such package also cannot be built: On FreeBSD there are 3,559 packages (including variants and metapackages) available from the repository at the present time. In the Solaris repo it’s only 2,851 packages. That’s certainly a nice start – but don’t expect to run a full-fledged desktop (or even X11 at all) there, yet!

In Linux land, distributions that come with glibc version 2.23 or newer work best. On distributions with older glibc versions (e.g. CentOS 7), software will not run as the standard C library is missing some required symbols. Raven will need to be bootstrapped again to support those distros. This is likely to happen before too long, but we’re not there, yet.

Current Firefox ESR version (+ sakura and pcmanfm in the panel)

MacOS (which might be supported soon), OpenBSD and NetBSD are not currently supported, nor is Linux with musl-libc or μclibc. Also currently Raven is amd64 only. ARM64 support is planned and i386 might or might not happen but are not available now.

Current status

At this time Raven is probably most interesting for people who love tech and enjoy tinkering on *nix systems as well as those who like the features and are ok with being early adopters. Yes, in general it’s ready for the latter. At least two people (including me) use Raven’s packages exclusively on one of their machines. I’d say it is ready as a daily driver (if you can live with the limited set of software available – or consider adding more ports). In fact I built a laptop that I use e.g. for on-call duty with it. Since that one is critical, it probably needs to be considered as “in production use”.

It’s possible to install various text mode applications with Raven, but X11 is also available. You can choose from multiple window managers or from at least two desktop environments (Lumina and the ultra-light EDE). Xfce4 is partially available (i.e. the panel is already ported). If you’re looking for web browsers, a current version of Firefox ESR (called “rustless-firefox”) can be installed as well as Surf, a simple webkit-based browser. The LibreOffice suite is available in its latest version, too. The same is true for the just released Perl 5.28 and Python 3.7.

Running Chocolate DooM and Chocolate Heretic

Oh, and if you’re into gaming… It’s not all just serious stuff. Yes, you can install and play DooM!

Conclusion

Ravenports is a fascinating project with lots and lots of possibilities. I wanted to get into porting with FreeBSD for quite a while but hesitated as I’m not a programmer. Then again I had been interested in package building for a long time and had played around with it on Arch Linux quite a bit. After my submissions to FreeBSD had been rotting in bug tracker for months (and still are after almost a year), I chose to give Raven a try in the meantime.

I was already familiar with Pkg and had used Synth before, too. Bootstrapping Raven’s pkg and then installing stuff was as easy as expected. The same was true for building the ports myself. Then I did quite a bit of reading and wrote my first port. It didn’t take more than 5 minutes after I opened my pull request on GitHub, before John responded – and the port was committed not much later. This was such a huge contrast that I decided to do more with Raven.

There was a learning curve, yes, but I received lots of help in getting started. I obviously liked the project enough to become a regular contributor and even got commit access to the ravensource repo later. Currently I’m maintaining just over 80 ports and I hope to write many more in the future. There have been some hard ports along the way (where I learned a lot about *nix), but lots of things are actually pretty easy once you get the hang of it.

Tongue-in-cheek: Make chaos or “make sense”!

If this post got you interested, just give it a try. Feel free to comment here and if you run into problems I’ll try to help. After this general overview of Raven the next post I plan to write will be on actually using it.

Modern-day package requirements

A little rant first: Many thanks to the EU (and all the people who decide on topics related to tech without having any idea on how tech stuff actually works). Their GDPR is the reason for me having been really occupied with work this month! Email being a topic that I’m teaching myself while writing the series of posts about it, I have to get back to it as time permits. This means that for May I’m going to write about a topic that I’m more familiar with.

Benefits of package management

I’ve written about package management before, telling a bit about the history of it and then focusing on how package management is done on FreeBSD. The benefits of package management are so obvious that I don’t see any reason not to content myself with just touching them:

Package management makes work put into building software re-usable. It helps you to install software and to keep it up to date. It makes it very easy to remove things in a clean manner. And package management provides a trusted source for your software needs. Think about it for just a moment and you’ll come up with more benefits.

Common package management requirements

But let’s take a look at the same topic from a different angle. What do we actually require our package systems to do? What features are necessary? While this may sound like a rather similar question, I assure you that it’s much less boring. Why? Because we’re looking at what we need – and it’s very much possible that the outcome actually is: No, we’re not using the right tool!

Yes, we need package management, obviously. While there’s this strange, overly colorful OS that cannot even get the slashes in directories right, we can easily dismiss that. We’re talking *nix here, anyway!

Ok, ok, there’s OmniOS with its KYSTY policy. That stands for “keep your software to yourself” and is how the users of said OS describe the fact that there’s no official packages available for it. While it’s probably safe to assume that the common kiddies on the web don’t know their way around on Solaris, I’m still not entirely convinced that this is an approach to recommend.

Going down that road is a pretty bold move, though. Of course it’s possible to manage your software stack properly. With a lot of machines and a lot of needed programs this will however turn into an abundance of work (maybe there are companies out there who enjoy paying highly qualified staff to carefully maintain software while others rarely spend more than a couple of minutes per day to keep their stuff up-to-date).

Also if you’re a genius who uses the method that’s called “It’s all in my head!” in the Linux from Scratch book, I’m not going to argue against it (except that this is eventually going to fail when you have to hand things over to a mere mortal when you’re leaving).

But enough of those really special corner cases. Let’s discuss what we actually require our package systems to provide! And let’s do so from the perspective not of a hobby admin but from a business-orientated one. There are three things that are essential and covered by just about any package system.

Ease of use

One of the major requirements we have today is that package management needs to be easy to use. Yes, building and installing software from source is usually easy enough on *nix today. However figuring out which configure options to use isn’t. Build one package without some feature and you might notice much later that it’s actually needed after all. Or even find that you compiled something in that’s getting in the way of something else later! Avoiding this means having to do some planning.

Reading (and understanding!) the output of ./configure –help probably isn’t something you’re going to entrust the newly employed junior admin with. Asking that person to just install mysql on the new server will probably be ok, though. Especially since package managers will usually handle dependencies, too.

Making use of package management means that somebody else (the package maintainer) has already thought about how the software will be used in most of the cases. For you this means that not having to hire and pay senior admins for work that can be done by a junior in your organization, too.

Fast operations

Time is money and while “compiling!” is a perfectly acceptable excuse for a dev, it shouldn’t be for an admin who is asked why the web server still wasn’t deployed on the new system.

Compiling takes time and uses resources. Even if your staff uses terminal multiplexers (which they should), thus being able to compile stuff on various systems at the same time, customers usually want software available when they call – and not two hours later (because the admin was a bit confused with the twenty-something tmux sessions and got stuck with one task while a lot of the other compile jobs have been finished ages ago).

Don’t make your customers wait longer than necessary. Most requests can be satisfied with a standard package. No need to delay things where it doesn’t make any sense.

Regular (security) updates

It’s 2018 and you probably want that new browser version that mitigates some of the Spectre vulnerabilities on your staff’s workstations ASAP. And maybe you even have customers that are using Drupal, in which case… Well, you get the point.

While it does make sense to subscribe to security newsletters and keep an eye on new CVEs, it takes a specialist to maintain your own software stack. When you got word of a new CVE for a program that you’re using that doesn’t mean the way you built the software makes it vulnerable. And perhaps you have a special use-case where it is but the vulnerability is not exploitable.

Again this important task is one that others have already done for you if you use packaged software from a popular repository. Of course those people are not perfect either and you may very well decide that you do not trust them. Doing everything yourself because you think you can do better is a perfectly legitimate way of handling things. Chances are however that your company cannot afford a specialist for this task. And in that case you’re definitely better off trusting the package maintainers than carelessly doing things yourself that you don’t have the knowledge for.

Special package management requirements

Some package managers offer special features not found in other ones. If your organization needs such a feature this can even mean that a new OS or distribution is chosen for some job because of that. Also repositories vary greatly in the number of software they offer, in the software versions that they hold and in the frequency of updates taking place.

“Stability” vs. “freshness”

A lot of organizations prefer “stable”, well-tested software versions. In many cases I think of “stable” as a marketing word for “really old”. For certain use-cases I agree that it makes sense to choose a system where not much will change within the next decade. But IMO this is far less often the case than some decision makers may think.

The other extreme is rolling-release systems which generally adapt the newest software versions after minimal testing. And yes, at one point there was even the “Arch server project” (if I remember the name correctly), which was all about running Arch Linux on a server. In fact this is not as bad an idea as it may seem. There are people who really live Arch and they’ll be able to maintain an Arch server for you. But I think this makes most sense as a box for your developers who want to play with new versions of the software that you’re using way before it hits your actual dev or even prod servers.

Where possible I definitely favor the “deliver current versions” model. Not even due to the security aspect (patches are being backported in case of the “stable” repositories) but because of the newer features. It’s rather annoying if you want to make use of the jumphost ability of OpenSSH (for which a nice new way of doing it was introduced not too long ago) and then notice you can’t use it because there’s that stupid CentOS box with its old SSH involved!

Number of packages

If you need one or a couple of packages that are not available (or too old) in the package repository of your OS or distribution, chances are that external repos exist or that the upstream project provides packages. That may be ok. However if you find that a lot of the software that you require is not available this may very well be a good reason to think about using a different OS or distribution.

A large number of packages in the repository increases the chance that you may get what you need. Still it can very well be the case where certain packages that you require (and which are rather costly to maintain yourself) are available on another repo.

Package auditing

Some package systems allow you to audit the installed packages. If security is very important for your organization, you’ll be happy to have your package tool recommend to “upgrade or deinstall” the installed version of some application because it’s known to be vulnerable.

Flexibility

What if you have special needs on some servers and require support for rarely needed functionality to be compiled into some software? With most package systems you’re out of luck. The best thing that you can do is roll your own customized package using a different name.

The ports tree on *BSD or portage on Gentoo Linux really show their power in this case, allowing you to just build the software easily and with the options that you choose.

Heterogeneous environments

So most of the time it makes perfect sense to stick to the standard repository for your OS or distribution. If you have special needs you’d probably consider another one and use the standard repo for that one. But what about heterogeneous environments?

Perhaps your database product only runs on, say, CentOS. You don’t have much choice here. However a lot of customers want their stuff hosted on Linux but they demand newer program versions. So a colleague installed several Ubuntu boxes. And another colleague, a really strange guy, slipped in some FreeBSD storage servers! When the others found out that this was not even Linux and started protesting (because “BSD is dying”), they were already running too damn well to replaced with something that does not have as good ZFS support.

A scenario like that is not too uncommon. If you don’t do anything about it, this might lead to “camps” among the employees; some of them are sure that CentOS is so truly enterprise that it’s the way to go. And of course yum is better than apt-get (and whatever that BSD thing offers – if anything). Some others laugh at that because Ubuntu is clearly superior and using apt-get feels a lot more natural than having to use yum (which is still better than that BSD thing which they refuse to even touch). And then there’s the BSD guy who is happy to have a real OS at his hand rather than “kernel + distro-chosen packages”.

In general if you are working for a small organization, every admin will have to be able to work with each system that is being used. Proper training for all package systems is probably expansive and thus managers will quite possible be reluctant to accept more than two package systems.

Portability

There’s a little known (in the Linux community) solution to this: Pkgsrc (“package source”). It’s NetBSD’s package management system. But with probably the most important goal of the NetBSD project being portability, it’s portable, too!

Pkgsrc is available for many different platforms. It runs on NetBSD, of course. But it runs on Linux as well as on the other BSDs and on Solaris. It’s even available for commercial UNIX platforms and various exotic platforms.

For this very nature of it, Pkgsrc may be one answer for your packaging needs in heterogeneous environments. It can provide a unified means of package management across multiple platforms. It rids you of the headache of version jungle if you use different repositories for different platforms. And it’s free and open source, too!

Is it the only solution out there? No. Is it the best one? That certainly depends on what you are looking for specifically. But it’s definitely something that you should be aware of.

What’s next?

The next post will be about a relatively new alternative to traditional package management systems that tries to deliver all the strong points in one system while avoiding their weaknesses!