Illumos (v9os) on SPARC64 SunFire v100

Over the last month or so I’ve written a couple of articles on an old SunFire v100 machine that I own for a while now. First I took a look at the hardware of the machine and the LOM (Lights Out Management). Then I installed OpenBSD 6.0 from CD and updated all the way to 6.5. Finally I played a bit with OpenBSD to see what it can do and how well it supports SPARC64. This post will be the last SPARC64 one before I visit other topics again.

v9os?

While I was pretty happy with OpenBSD on the SunFire, there’s one reason that I wanted to try out something else, too. That reason has three letters: Z-F-S. The first thing that I tried out when I got the hardware, was FreeBSD – but I ran into problems. I’ve managed to overcome circumvent them (might be worth another story in the future), only to find that FreeBSD does not support ZFS on SPARC64!

One option that suggests itself, is just putting Solaris on there. I have a copy of Solaris 10 for Sparc, but I prefer to keep things Open-Source. Also there’s the problem, that my machine is old enough to not have a DVD drive and it doesn’t support booting from USB and the like.

So it’s illumos. Since I’m really just getting started with the broader Solaris universe, I had to do a little research first. And I was a little surprised that most illumos distros seem to not even support Sparc at all! Of the four that do

  • OpenSXCE seems dead (last release in 2014)
  • DilOS uses Debian packaging (which is not my cup of tea at all)
  • Tribblix sounds really interesting to me, but does not fit on a CD
  • v9os is a minimal Sparc distro that is small enough

As you can see, there wasn’t so much choice after all! While v9os is an experimental one-man project that you should probably stay away from for production use, it might be just right for my purposes of tinkering with an old machine.

Installing the OS – first try

There are not many preparations necessary: I downloaded the ISO image and burned it on a CD. Then I connected to my SunFire via serial, powered it on and put the CD into the drive. It takes quite some time, but after a while I can read that v9os is in fact starting.

Booting up v9os from the CD

After the system booted, it gives the user the option to select a keymap.

Keymap selection

Then it shows the installation menu. There you can choose if you want to install, load additional drivers, drop to a shell, change the terminal type or reboot. I go with the first option.

v9os installation menu

After a moment the installer has started an a welcome screen is printed. Unfortunately in my case there’s a problem with the CD, so that four lines of debug info overwrite important information: How to actually proceed with the installation! But this is an OpenSolaris derivative, and so it’s not that hard to figure out that F2 is the key to go on.

v9os installer: Welcome screen

Next it’s selecting the disk to install on. I thought that it all looked good – and didn’t pay much attention to the message “A VTOC label was not found.”. VTOC is the Volume Table Of Contents, the SPARC partition scheme (think MBR/GPT on amd64). We’ll come back to that a little later. 😉

v9os installer: Disk selection

I think that the installer is quite nice. It even offers help pages that give newcomers like me an idea of what they should do for the current step. Great work on that!

v9os installer: Disks help page

Then you can choose to either dedicate the whole disk to v9os or just use a slice. I decide to go the easy route and select the former.

v9os installer: Disk layout selection

Now the installer wants to know the hostname for the new system. The suggested default of v9os is fine for me since I don’t plan to add another machine with that OS to my network anytime soon.

v9os installer: Hostname selection

Finally you can select the time zone – or rather: the zone region.

v9os installer: Time zone selection

Unfortunately things went sideways after that choice and I had to reset the machine…

Ok, after going through the previous steps again, I decided to give the advanced setup a try and selected slicing up the drive.

v9os installer: Slice selection

Unfortunately the result was the same as before: The installer just died. I tried again a few times, playing with different slice setup, but didn’t have any luck.

The installer died… Time to reboot.

At this point I was out of ideas on what else I could try, so I removed the CD and powered down the system.

Writing the label manually

When I powered the system on again, I had forgotten that I removed the CD and to my surprise OpenBSD (the system that I had previously installed on the machine) booted up! This meant that the installer had not even changed anything on the disk, yet!

My next guess was (and still is) that the v9os installer might have problems with BSD disklabels being present on the drive. I took a look at the disklabel from OpenBSD, just to find out some information about the drive.

OpenBSD’s disklabel information of the system hard drive

Then I booted the v9os install medium again but this time selected the shell option. After a little research I found out how to get some drive information on Solaris with iostat.

v9os shell session: Collecting drive hardware info

Next I decided to give the format utility a try. I don’t know if v9os stripped down some hardware information and that together with the disk being really old, it wasn’t properly auto-detected. So I had to do something that I haven’t done in years (and never missed it): Typing in the geometry information by hand!

Typing in disk geometry information (Ah, the (bad!) memories…)

Once the drive has been described to the utility, it shows a menu of what it can do. I haven’t used that program before and judging from the name alone was a bit surprised at how powerful it seems to be. Things like being able to define profiles must have been pretty useful in the past.

Solaris’ format utility

Since I want to partition the drive, I select that. I’m presented with a sub-menu, giving me some more choices.

Partitioning menu of format

I have no clue what a Solaris partitioning scheme should look like (need to explore some older versions of that OS somewhen!).

Partitioning the drive for Solaris

So I look around a little but eventually accept the proposed default and just hope that this works.

Installing the OS – second try

After restarting the machine again and choosing the installer, it looks like this time there is no missing disklabel. At least! But will it make a difference?

Returning to the installer: Partitioning was detected

And yes! Now the installer continues and gets the data written to disk!

Finally installing the OS!

The process takes quite a while – but that’s due to the slow machine that I’m using. Eventually the installation is finished.

v9os installer: All done!

First steps with v9os

Another reboot and after removing the CD-ROM from the drive, the freshly installed system boots up. A moment later it displays the prompt where I can log in using the user root and the password solaris.

First start of v9os

The first thing that I want to do is to get rid of the serial console. So I set up networking and enable SSH.

Setting up networking and enabling SSH

Then I disable the automounter to make the home directory writable and create a user for remote SSH login. Finally I enable the machine to do name resolution and give the new user a password.

Adding a user and name resolution capabilities

That should suffice to SSH into the box from another machine.

Package management with IPS

Logging in remotely works just fine. As v9os does not have an online package repository, I have to download a compressed copy of the repository from SourceForge.

SSHing into the v9os box and downloading the package repository

I don’t know much about the IPS package system and thus really struggle to make it all work. There is no guide on the v9os site and so I try to put the downloaded file in various locations, decompress it and try everything again. Since that also doesn’t work, I unpack the contents of the archive but still cannot get it right…

Struggling to get the repo working…

After more than an hour of struggling with pkg, reading manpages, doing online research and trying to fit everything together, I finally manage to remove the default publisher that comes with the system and add a new one that eventually works!

Finally figured out how to deal with IPS publishers

The v9os operating system is one of the strangest Unices that I’ve ever touched in not providing the vi editor with the system! But now that I have the repository available, I can simply install vim to find out that using packages does work after all.

Installing packages (vim) works!

This is about how far I wanted to take this quick post on v9os. If I had a faster machine, I might have been tempted to try and build the system from source. But with my old SunFire… No.

While v9os might not be fit for production use, I accomplished one goal over OpenBSD: I have an operating system on the machine that is installed on ZFS!

ZFS on SPARC64 with v9os

Conclusion

The v9os operating system is an exotic one for sure. But it’s nice to see that somebody values SPARC64 machine and illumos enough to put the time required to built something like this into such a project. And actually I think it’s not half bad! I didn’t do too much with it, but it seemed stable and except for the installer problem (it would probably just have worked on an empty drive) everything worked fine.

Well, maybe some hints on how to get the package repo in place would have saved me some time… On the other hand Solaris veterans are likely to get it working with just a few commands. And while it has been kind of frustrating for a while, it has also lead to at least a basic understanding of what IPS is and how it works. I’m sure that I’d have missed at least some of that if I had just copied some lines from a guide.

I might not end up making v9os my primary operating system (for various obvious reasons). But it’s another nice little part in the mosaic of the illumos world that I’ve started exploring. Also I noticed that I’ve become a little bit more comfortable with using an OpenSolaris-derivative. Compared to my first encounter with OmniOS, it didn’t take me as long to figure out the very basics again. Which is always a good sign.

A SPARC in the night – SunFire v100 exploration

While we see a total dominance of x86_64 CPUs today, there are at least some alternatives like ARM and in the long run hopefully RISC-V. But there are other interesting architectures as well – one of them is SPARC (the Scalable Processor ARChitecture).

This article is purely historic, I’m not reviewing new hardware here. It’s more of a “20 years ago” thing (the v100 is almost that old) written for people interested in the old Sun platform. The intended audience is persons who are new to the Sun world, who are either to young like me (while I had a strong interest in computers back in the day, I hadn’t even finished school, yet, and heck… I was still using Windows!) or never had the chance to work with that kind of hardware in their professional career. Readers who know machines like that quite well and don’t feel like reading this article for nostalgic reasons might just want to skip it.

The SPARC platform

SPARC is a Reduced Instruction Set Computing (RISC) Instruction Set Architecture (ISA) developed by Sun Microsystems and Fujitsu in 1986. Up to the Sun-3 series of computers, Sun had used the m68k processors but with Sun-4 started to use 32-bit SPARC processors instead. The first implementation is known as SPARCv7. In 1992 Sun introduced machines with v8, also known as SuperSPARC and in 1995 the first processors of SPARCv9 became available. Version 9, known as UltraSPARC, is a 64-bit architecture that is still in use today.

SunFire v100: Top and front view

SPARC is a fully open ISA, taken care of by SPARC International. Architecture licenses are available for free (only an administration fee of 99$ has to be payed) and thus any interested corporation could start designing, manufacturing and marketing components conforming to the SPARC Architecture. And Sun did really mean it with OpenSPARC: They released the Verilog code for their T1 and T2 processors under the GPLv2, making them the first ever 64 bit processors that were open-sourced. And not enough with that – they also released a lot of tools along with it like a verification suite, a simulator, hypervisor code and such!

After Sun was acquired by Oracle in 2010, the future of the platform became unclear. Initially, Oracle continued development of SPARC processors, but in 2017 completely terminated any further efforts and laid off employees from the SPARC team.

Fujitsu has made official statements that they are continuing to develop the SPARC-based servers and even about a “100 percent commitment”. In the beginning of this year, they even wrote about a Resurgence of SPARC/Solaris on the company’s blog and since they are the last one to provide SPARC servers (which are still highly valued by some customers), chances are that they will continue improving SPARC. According to their roadmap, even a new generation is due for 2020.

So while SPARC is not getting a lot of attention these days, it’s not a dead platform either. But will it survive in the long run? Time will tell.

SunFire v100

I’m working for company that offers various hosting services. We run our own data center where we also provide colocation for customers who want that. Years ago a customer ran a root server with an (now) old SunFire v100 machine. I don’t remember when it was decommissioned and removed from the rack, but that must have been quite a while ago.

SunFire v100: Back view

That customer was meant to come over to collect the old hardware and so we put the machine in the storage room. For whatever reason, he never came to get it. Since it had been sitting there for years now, I decided to mail the customer and asked if he still wanted the machine. He didn’t and would in fact prefer to have us to dispose of it. So I asked if he’d be ok with us shreddeing the hard drives and me taking the actual machine home. He didn’t have any objections and thus I got another interesting machine to play with.

The SunFire v100 is a 1U server that was introduced in 2001 and went EOL in 2006. According to the official documentation, the machine came with 64 bit Solaris 8 pre-installed. It was available with an UltraSPARC IIe or IIi processor and had a 40 GB, 7200 RPM IDE HDD built-in. My v100 has 1GB of RAM and a 550 MHz UltraSPARC IIe. I also put a 60 GB IBM HDD into it.

It has a single PDU, two ethernet ports as well as two USB ports. It also features two serial ports – and these are a little special. Not only are they RJ-45, but they have two different uses cases. One is for the LOM (we’ll come to that a little later), the other one is a regular serial port that can be used e.g. to upload data uninterrupted (i.e. not going to be processed by the LOM). The serial connection uses 9600 baud, no parity, one stop bit and full duplex mode.

RJ-45 to DB9 cable and DB9 to USB cable

The other interesting thing is the system configuration card. It stores host ID and MAC address of the server as well as NVRAM settings. What is NVRAM? It’s an acronym for Non-Volatile Random-Access Memory, a means for storing information that must not be lost when the power goes off like regular RAM does. If you’re thinking “CMOS” in PC terms, you’re right – except it seems that Sun used a proper means of NVRAM and not an in fact volatile source made “non-volatile” by keeping the data alive with the help of a battery. The data is stored on a dedicated chip, or in this case on a card. The advantage of the latter is that it can be easily transferred to another system, taking all the important configuration with it! Pretty neat.

Inside the v100

When I opened up the box, I was actually astonished by how much space there was inside. I know some old 1U x86 servers from around that time (or probably a little later) that really are a pain to work with. Fitting two drives into them? It’s sure possible, but certainly not fun at all. At least I hated doing anything with them. And those at least used SATA drives – I haven’t seen any IDE machines in our data center, not even with the oldest replacement stuff (it was all thrown out way before I got my job). But this old Sun machine? I must say that I immediately liked it.

SunFire v100: Inside view

Taking out the HDD and replacing it with another drive was a real joy compared to what I had feared that I’d be in for. The drive bays are fixed using a metal clamp that snaps into a small plastic part (the lavender ones in the picture). I’ve removed the empty bay and leaned it against the case so that it’s easier to see what they look like. It belongs where the ribbon cable lies – rotated 90 degrees of course.

Old x86 server for comparison – getting two drives in there is very unpleasant to do…

All the other parts are easily accessible as well: The PDU in the upper left corner of the picture, the CDROM drive in the lower right, as well as the RAM modules in the lower left one. It’s all nicely laid out and well assembled. Hats off to Sun, they really knew what they were doing!

Lights out!

I briefly mentioned the LOM before. It’s short for Lights-Out Management. You might want to think IPMI here. While this LOM is specific to Sun, its basic idea is the same as the wide-spread x86 management system: It allows you do things to the machine even when it’s powered off. You can turn it on for example. Or change values stored in the NVRAM.

LOM starting up

How do we access it? Well, the machine has a RJ-45 socket for serial connections appropriately labeled “LOM”. The server came with two cables to use with it, one RJ-45 to DB26 (“parallel port”) used with e.g. a Sun Workstation, and a RJ-45 to DB9 (“serial port” a.k.a. “COM port”). Then you can use any of the various tools usually used for serial connections like cu, tip or even screen.

Just plug your cable into say your laptop and the other end into the A/LOM port, then you can then access the serial console. If you plug in the power cable of the SunFire machine now, you will see the LOM starting up. Notice that the actual server is still off. It’s in standby mode now but the LOM is independent of that.

LOM help text

By default, the LOM port operates in mixed mode, allowing to access both the LOM and the serial console. These two things can be separated if desired; then the A port is dedicated to the LOM only and the console can be accessed via the B port.

In case you have no idea how to work with the LOM, there’s a help command available to at least give you an idea what commands are supported. Most of these commands have names that make it pretty easy to guess what they do. Let’s try out some!

LOM monitoring overview (powered off)

Viewing the environment gives some important information about the system. Here it reveals that ALARM 3 is set. Alarm 1, 2 and 3 are software flags that don’t do anything by themselves. They can be set and used by software installed on the Solaris operating system that came with the machine.

I really have no idea why the alarm is set. It was that way when I got the server. Even though it’s harmless, let’s just clear it.

Disabling alarm, showing users and booting to the ok prompt

The LOM is pretty advanced in even supporting users and privileges. Up to four LOM users can be created, each with an individual password. There are four privileges that these can have: A for general LOM administration like setting variables, U for managing LOM users, C to allow console access as well as R for power-related commands (e.g. resetting the machine). When no users are configured, the LOM prompt is not protected and has full privileges.

OpenBoot prompt

It is also possible to set the boot mode in the LOM. By doing this, the boot process can e.g. be interrupted at the OpenBoot prompt which (for obvious reasons) is also called the ok prompt. In case you wonder why the command is “boot forth” – this is because of the programming language Forth which the loader is written in (and can be programmed in).

ok prompt help

In the ok prompt you can also get help if you are lost. As you can see, it is also somewhat complex and you can get more help on the respective areas that interest you.

Resetting defaults and probing devices

OpenBoot has various variables to control the boot sequence. Since I got a used machine, it’s probably a good idea to reset everything to the defaults.

From the ok prompt it’s also possible to probe for devices built into the server. In this case, an HDD and a CDROM drive were found which is correct.

Setting NVRAM variables, escaping to LOM, returning to the ok prompt and resetting the machine

The ok prompt allows for setting variables, too, of course. Here I create an alias for the CDROM drive to get rid of working with the long and complex device path. Don’t ask me about the details of the latter however. I found this alias on the net and it worked. I don’t know enough about Solaris’ device naming to explain it.

Next I set the boot order to CDROM first and then HDD. Just to show it off here, I switch back to the LOM – using #. (hash sign and dot character). That is the default LOM escape sequence, however it can be reconfigured if desired. In the LOM I use the date command to display how long the LOM has been running and then switch back to the ok prompt using break.

LOM monitoring overview while the machine is running

Finally I reset the machine, so that the normal startup process is initiated and an attempt at booting from the CDROM is being made. I threw in a FreeBSD CD and escaped to the FreeBSD bootloader (which was also written in Forth until it was replaced with a LUA-based one recently).

Showing the monitoring overview while the machine is actually running is much more interesting of course. Here we can see that all the devices still work fine which is great.

LOM log and date, returning to console and powering off

Finally I wanted to show the LOM log and returning to the console. The latter shows the OK prompt now. Mind the case here! It’s OK and not ok. Why? Because this is not the OpenBoot prompt from the SunFire but the prompt from the FreeBSD loader which is the second-stage loader in my case!

That’s it for the exploring this old machine’s capabilities and special features. I just go back to the LOM again and power down the server.

Conclusion

The SunFire v100 is a very old machine now and probably not that useful anymore (can you say: IDE drive?). Still it was an interesting adventure for me to figure out what the old Sun platform would have been like.

While I’m not entirely sure if this is useful knowledge (SPARC servers in the wild are more exotic then ever – and who knows what the platform has evolved into in almost 20 years!), I enjoy digging into Unix history. And Sun’s SPARC servers are most definitely an important mosaic in the big picture!

What’s next?

Reviewing this old box without installing something on there would feel very incomplete. For that reason I plan to do another article about installing a BSD and something Solaris-like on it.

Rusted ravens: Ravenports march 2019 status update

It’s been a couple of months since I last wrote about Ravenports, the universal *nix application building framework. Not exactly being a slowly moving project, a lot has happened since then.

Platform support

Raven currently supports DragonFly BSD, FreeBSD, Linux and Solaris/Illumos, the latter being only in the form of binary packages (except for when you have access to an installation of Solaris 10u8 – which can be used to build packages, too).

People following the project will notice the lack of macOS/Darwin support mentioned here. This is not a mistake as support for that platform has been put on hold for now. While Raven has successfully been bootstrapped on macOS before, the developers have lost access to any macOS machines and thus cannot continue support for it.

This does not mean that platform is gone forever. It might be resurrected at a later point in time if given access to a Mac again. The adventurous can even try to bootstrap Raven on different platforms now as the process has been documented (with macOS as the example).

I intended to do some work on bootstrapping Raven on FreeBSD/ARM64 – only to find that FreeBSD unfortunately still has a long way before making that platform tier 1. At work I had access to server-class ARM64 hardware, but current versions of FreeBSD have trouble booting up and I could not get the network running at all (if you’re interested in the details see my previous post). I’m still hoping for reactions on the mailing list but until upstream FreeBSD is fixed on ThunderX trying to bootstrap does not make much sense.

Toolchain and package updates

The toolchain used by Ravenports has been updated to GCC 8.3 and Binutils 2.32 on all four supported platforms (access to Mac was lost before the toolchain update).

As usual, Solaris needed a bit of extra treatment but up to date compiler and tools are available for it now, too. Even better: The linker from the LLVM project (lld) is available and usable on Solaris/Illumos now as well. Since it takes several hours (!) to link on Solaris, a mostly static lld executable was added to the sysroot package for that platform. This way this long-building package does not have to be rebuilt as often.

Packages have been rebuilt with this bleeding-edge toolchain (plus the usual fallout has been collected and fixed). So if you are using Raven, you are making use of the latest compiler technology with the best in optimization. Of course a lot of effort went into providing the most current versions of the packaged software, too (at least where that is feasible).

On the desktop side of things I’ve added the awesome window manager to the list of available software. It’s actually my WM of choice, but not too many people are into tiling so I postponed this one for after making Xfce available. Work on bringing in more Lua-related ports for an advanced configuration it is ongoing, but the WM is already usable as it is now.

I’ve also done a bit of less visible work, going back to many ports that I created previously and added in missing license info. This work is also not completed, yet, but the situation is improving, of course.

Rust!

One of the big drawbacks of Ravenports as stated last time, was the lack of the Rust compiler. This effectively meant a showstopper for things like current versions of Firefox, Thunderbird, librsvg, etc. The great news is that this blocker has been mostly removed: Rust is available via Raven for Dragonfly, FreeBSD and Linux! Solaris/Illumos support is pending, I think that any helping hand would be greatly appreciated.

Bringing in Rust was a big project on its own. Adding an initial bootstrap package for Dragonfly alone took months (thank you, Mr. Neumann!). The first working version of the port made Rust 1.31 available. It has since been updated to version 1.32 and 1.33 and John has added functionality to the Raven framework to work with Rust’s crates as well as scripts to assist with future updates. Taking all of that into consideration, Rust support in Raven is already pretty good for the short time that we have it.

Eventually even a port for Firefox landed – as of now it’s marked broken, though. The reason is that while it does compile just fine, the program crashes when actually started. The exact cause for this is yet unknown. If anybody with some debugging abilities has a little time on his hands, nailing down what happens would be a task that a lot of people will be benefit from for sure!

Updated ravenadm

Ravenadm, the Ravenports administration tool, has seen several updates with new features. Some have brought internal changes or new features necessary for new or updated packages. One example is a project-wide fix for ports that build with Meson: Before the change many programs needed special treatment to make Meson honor the rpath for the resulting binaries. Now Raven can automatically take care of this, saving us a whole bunch of sed commands in the specification file. Another new feature is the “Solaris functions” mechanism which can automatically fix certain functions that required generating patches before. Certainly also very nice to have!

Probably my favorite new feature is that Ravenadm now supports concurrent processes in several cases: While you cannot start a second set of package builds at the same time for obvious reasons, it is now possible to ask Ravenadm in which bucket a certain port lives, sort manifests, and such while building packages! I cannot say how much the previous behavior got in my way while doing porting work… This makes porting much, much more pleasant.

A last improvement that I want to mention here is a rather simple one – however one that has a huge impact. Newer versions of Ravenadm put all license-related texts into the logs! This means you can simply look at the log and see if e.g. the terms got extracted correctly. Before you had to use the ENTERAFTER option to enter an interactive build session and look at the extracted file. This is a huge improvement for porters.

SSL

Another big and most likely unique feature added to Raven recently is SSL autoselection. Raven has had autoselection facilities for Python, Ruby and Perl for about a year now. The latter allow for multiple versions of the interpreters to be installed in parallel and take care of calling the actual binary with the same parameters, preferring the newest version over older ones (until configured differently).

Raven supports LibreSSL, OpenSSL as well as LibreSSL-devel and OpenSSL-devel. Before the change, you could select the SSL library to use in the profile and it would be used to link all packages against it. Now we have even more flexibility: You can e.g. build all the packages against LibreSSL by default and just fall back to OpenSSL for the few packages that really require it!

And in fact Raven takes it all one step further: You can have OpenSSL 1.0.2 and OpenSSL 1.1.1 (which introduced braking changes) installed in parallel and use packages on the same system where some require the new version and some that cannot use it, yet! Pretty nice, huh?

Future work

Of course there are still enough rough edges that require work. Probably the most pressing issue is to get Firefox working so Raven’s users can have access to a convenient and modern browser. There are also quite some programs which need ports created for them. The goal here is to provide the most critical programs to allow Dragonfly to make the switch from Dports to Ravenports for the official packages.

On FreeBSD Filezilla does not currently work: It cannot be built with GCC due to a compiler bug in GCC 7.x and 8.x. Therefore it is a special port that get’s build with Clang instead. The problem is that libfilezilla needs to be built with the same toolchain – and that cannot currently be built with Clang using Raven…

Raven on Linux has some packages not available due to additional dependencies on that platform. I begun adding some Linux-specific ports but lost motivation to do so pretty fast (there are enough other things after all). Also the package manager is still causing pain, randomly crashing.

Solaris is also missing quite some packages. This is due to additional patches being required for a lot of software to build properly. Ravenports tries to support this platform as good as possible; however this could surely be improved if anybody using Solaris or an Illumos distribution as his or her OS of choice would start using Raven and giving feedback or even contribute.

Get in touch!

Interested in Raven? Get in touch with us! There is an official IRC channel now (#ravenports on Freenode) which is probably the best place to talk to other Raven users or the porters and developers. You can of course also send an email.

If you want to contribute, there is now a new “Customsource” repository on GitHub that you can create pull requests against. Feel free to contribute anything from finished ports that might only need polish to WIP ports that are important for you but you got stuck with.

There are many other means of helping with the than project then doing porting work, though. Open issues if you find problems with packages or have an idea. Also tell us if you read the wiki and found something hard to understand. Or if you could use a tutorial for something – just ping me. Asking doesn’t hurt and chances are that I can write something up.

Got something else that I didn’t talk about here? Tell us anyway. We’re pretty approachable and less elitist than you might think people who work on new package systems would be! 😉

Ravenports explained: Why not just join XYZ?

As the year comes to an end, I’ve seen quite some interest in my previous post. There has been a question on Reddit what the benefit(s) of Raven over Pkgsrc might be and why the developers don’t simply join an existing effort instead of building something new.

I’ve touched on this topic about half a year ago, but I think the question is worth a detailed reply that fully covers both parts of it. So I’ll try to answer 1) why Ravenports exists in the first place and 2) what sets it apart from Pkgsrc and other ports systems.

Why maintain Ravenports instead of working on Pkgsrc?

Well, obviously because its author felt it was worthwhile to start and maintain the project! Of course that leads to another and more important question – why didn’t John Marino just join e.g. Pkgsrc instead? The answer to that is: Well… He did.

John got his NetBSD commit bit and became a Pkgsrc developer back in the day when DragonflyBSD still used Pkgsrc by default. He maintained a ton of ports there and made sure that other people’s ports still worked on DF after they had been updated. DragonflyBSD had been considered a first-class citizen by Pkgsrc. However there had been two big problems:

1) Being primarily a NetBSD project, Pkgsrc development takes place mostly on NetBSD of course. Things were tested on NetBSD and then committed. There was no testing done on the other supported platforms – which is a completely comprehensible decision given the amount of ports available and the number of supported platforms as well as the need to get software updated in a somewhat timely manner! However this lead to frequent breakage. A few suggestions that made sense from the Dragonfly perspective could not be agreed upon taking the whole of Pkgsrc into account. In the end the policy was: “If things in the tree break for your platform, go ahead and fix it.” So basically the answer to problem 1 was: “Throw more manpower at it.”

2) As the small project that DragonflyBSD is, there simply were not too many people available for this task however. In fact it was largely John alone who did most of the work with some help here and there. It’s impossible to spend resources that you don’t have available!

As you can see problem 1 causes problem 2 – and that one proved to be unfixable. Thus the problems with Pkgsrc grew and there was really not much that could have been done about it. And as the suggestions to somewhat relieve the worst impact were turned down, Dragonfly had to give up Pkgsrc. Please keep in mind that there’s a major difference between how Dragonfly used Pkgsrc and how some other platforms do. Sure, it’s great that you can use Pkgsrc on AIX to obtain some current software. Same thing for many other systems. Dragonfly used Pkgsrc just as NetBSD does, though: As the primary means to get software installed. Large-scale breakage of packages is a no-go in such a case, especially if it happens somewhat often and was bound to happen again and again.

Ok – another project then. Adapt the FPC maybe?

John then brought the new FreeBSD package manager as well as the FreeBSD ports collection over to Dragonfly with a system called “delta ports” or Dports. It’s basically an overlay with patches that Dfly requires to build those ports. Even though the FPC is meant for FreeBSD only and Pkgsrc – being cross-platform – might seem like the more logical candidate, this worked out a lot better and John maintained Dports for years.

In maintaining so many ports for both Pkgsrc and Dports he had a quite few ideas on how to do things better. They wouldn’t fit into the projects as they were organized, though. So he begun playing with various things on his own. Then… FreeBSD introduced flavored ports.

Don’t get me wrong here: I’m a FreeBSD user and I’m glad that flavored ports are finally available. However from a technical point of view they are implemented in a way that’s far from perfect. This is no wonder, though: When the ports tree was first introduced, nobody thought of flavors. What we have today is a fine example of a feature implemented as an afterthought. It works, yes, but it meant a disrupting change and broke expectations of all ports-related programs. It also made maintaining Dports much, much more time-intensive – to the point where it becomes no longer feasible to keep it up.

What does Ravenports have to offer over Pkgsrc?

Just like every younger project, Ravenports has the considerable advantage of starting fresh without the burden of choices that seemed right in the past but were probably regretted later. If this is combined with the will to learn from previous attempts to get packaging right as well as considerable experience with those, this has a lot of potential.

Think about it for a moment: FreeBSD’s ports collection shipped with the 1.0 release of the OS – and thus was created back in 1993. Pkgsrc began as a fork of it in 1997. So both were originally designed in a decade that has long passed (and in fact not even in this millennium!). Yes, both have been modernized over time. There are limits to this, however. It can be pretty hard to integrate new features into a structure that never meant to support anything like that. Do you think anybody in the mid 90’s could have thought about the needs of today? Ravenports deliberately does not support some old cruft. It’s meant for the coming decade of the 2020’s.

Here’s some strong points where Raven is ahead of Pkgsrc:

  • Tooling:
  • It offers a modern, integrated solution. There’s one control program (“ravenadm”) that deals with everything regarding Ravenports: It’s used to configure the package building system, it fetches the buildsheets (ports) and keeps them up to date, it builds all the packages or a subset thereof, …

  • Pristine package builds:
  • Everything is built in a chroot sandbox specifically assembled for that build process. There is no way that build dependencies clutter your build system (chances are you don’t want to use m4 or automake yourself and thus don’t need them installed on the OS). There’s also no way that installed packages of your system pollute the packages that Raven builds: The isolation prevents e.g. linking against additional stuff that you didn’t mean to.

  • It’s fast:
  • Did you ever run a bulk-build for Pkgsrc packages? Ravenports optimizes build times on modern systems by taking advantage of memory disks and such. The port scan alone makes a huge difference.

  • Potentially package manager agnostic:
  • Currently Raven supports only the Pkg package manager but as all it does is build packages, it was designed to support additional package managers if needed. You actually want it to generate rpm or pacman packages? Not currently implemented but certainly possible if desired.

  • Powerful default package manager:
  • Pkg, a modern tool for package management, is quite capable. If you read the manpages for it you will find out that it’s loaded with useful features. The old pkg_tools that Pkgsrc still use totally pale in comparison – and rightfully so.

  • Easy administration of multiple repos:
  • Need multiple repositories? No problem. Just create profiles for them. E.g. one that uses LibreSSL and another one that links against OpenSSL instead. Also you can choose the default version of Perl, Python, Ruby, ect. to use. And you can choose if MySQL should be Oracle’s MySQL, MariaDB, Galera, ect.

  • Convenient use of custom ports:
  • Can you use custom ports that are not in the official buildsheet collection? Sure thing. You can create directories for your custom ports and even use different ones in different profiles. Want to change an existing port? Just place one with the same name in your custom port directory and it will override the original one. Buildsheets from custom ports are generated automatically so there’s no hassle there. It probably doesn’t get much more convenient!

  • Variants and subpackages:
  • Package variants (i.e. “flavors”) and subpackages are not an afterthought and are thus used excessively right from the beginning. This makes package management with Raven very flexible.

  • Testing:
  • The Ravenports system has very strict rules for buildsheets. If the ravenadm tool considers a port to be valid, it is almost guaranteed that it is actually fine. Also packages can not only be mass-built but they can also be tested automatically as well (Is the RPATH ok? Are all required shared objects available? Is the manifest file complete? Are the required descriptions in place? Is the license ok or lacking? Things like that).

  • Automation:
  • Ravenports tries to automate many things that do not actually need human attention. For example quite often Python-related ports can be auto-generated. This saves time and effort of the maintainers that can be better spent on other things.

  • Modern day development:
  • Want to contribute something? It’s extremely easy. If you have a GitHub account you’re all set: Fork the git repo, make your changes, then commit and push them. Now all that’s left is opening a Pull Request. Yes, that’s all. If you don’t have a GH account, create one. Or send us patches as it was traditionally done. Ravenadm will happily create a template for you to assist you if you want to contribute a new port.

  • No ports ownership:
  • In Ravenports nobody “owns” a port. If you submitted one you become a contact for it. If somebody wants to make major changes to the port, that person is expected to contact you and communicate the proposals. Small or trivial changes however (like a simple version upgrade) can be done by anybody. This ensures rapid development and very fast adoption of new versions even if the original porter does not currently have the time to maintain everything in a timely manner.

  • Fast releases:
  • Ravensource provides new releases quite often. This way you can get pretty fresh software early on. There is no fixed time frame for it, though: Releases are made when it makes sense. If there have been major changes to the tree the next release might be delayed for testing.

  • Binary bootstrap:
  • Ravenports has a very simple and fast bootstrap process that makes use of binary packages for the respective platform. No system compiler required! Raven brings in its own full toolchain.

There are of course cases where it makes sense to use Pkgsrc and it’s not too hard to find any: E.g. if you need packages for a platform that’s unsupported in Raven or if you need software not yet available there. In the end this is Open Source: We’re all friends and using the right tool for the job makes sense.

Couldn’t Ports/Pkgsrc be modernized?

I’ve used Pkgsrc both in private and at work and I’m pretty happy that it’s available when I need it. But I don’t like the old pkg_tools much. They do their job but they are far from modern programs and really feel like relics today. And while I’m pretty happy with FreeBSD’s ports, those aren’t portable (and for some reason I’ve never been completely happy with Poudriere, FreeBSD’s package builder).

Before finally creating Ravenports, John wrote Synth, a very nice package builder for FreeBSD and DragonflyBSD that supports Ports/Dports. It has been put on hold in favor of Raven, but it is still maintained and I continue to use it on FreeBSD to build my packages.

John also created Pkgsrc-synth. It’s a version of Pkgsrc that uses the Pkg package manager. I’ve never tried it out – but it was stopped exactly two month ago as there seems to not have been any interest from the Pkgsrc people. I think this is a pitty, as pkg is really nice and has the right license for any BSD project. It could have been a chance to move Pkgsrc into a more modern direction. But meh.

Conclusion

Raven does not exist because everything else sucks. It exists because all the other candidates proved to not quite fit the needs of Ravenport’s author. As such it is a chance to keep the good parts of its various precursors that it heavily draws inspiration from. It’s a chance to combine these good parts to make something awesome. And it’s a chance to implement a lot of new ideas that should make sense in modern-day *nix package building which – for various reasons – cannot have a place in the old projects.

There’s still a lot of work to do, but we’re getting there. In my previous post I wrote that one of the big shortcomings was the lack of Rust. In the meantime Rust support has landed for DragonflyBSD, FreeBSD and Linux.

If there are any more questions feel free to post them here. I’m not on Reddit and I just saw the above question by accident. So I cannot promise to answer anywhere else than here.

Happy new year everyone!

One year of flying with the Raven: Ready for the Desktop?

It has been a little over one year now that I’m with the Ravenports project. Time to reflect my involvement, my expectations and hopes.

Ravenports

Ravenports is a universal packaging framework for *nix operating systems. For the user it provides easy access to binary packages of common software for multiple platforms. It has been the long-lasting champion on Repology’s top 10 repositories regarding package freshness (rarely dropping below 96 percent while all other projects keep below 90!).

For the porter it offers a well-designed and elegant means of writing cross-platform buildsheets that allow building the same version of the software with (completely or mostly) the same compile-time configuration on different operating systems or distributions.

And for the developer it means a real-world project that’s written in modern Ada (ravenadm) and C (pkg) – as well as some Perl for support scripts and make. Things feel very optimized and fast. Not being a programmer though, I cannot really say anything about the actual code and thus leave it to the interested reader’s judgement.

If you’re interested in a more comprehensive introduction to Ravenports, I’ve written one half a year ago.

Platforms

Ravenports has initially been developed on DragonFly BSD. When I became aware of it, it had already been ported to work on Linux, too. I liked the idea of the project, but had no DF or Linux boxes available for tinkering and didn’t feel like setting one up. Thus I moved on.

As I checked back a little later, FreeBSD support had been added. Since I had just lost my excuse not to try it out right away, I started playing with it – and was pretty happy. At that time I had trouble to get a port that I wrote into FreeBSD’s Ports Collection and thought that Raven could be an excellent playground to learn something and get a bit of experience that might help me later with FreeBSD.

The Xfce4 desktop – installed via Raven

I’ve long changed my mind, though! Raven is rather similar to FreeBSD’s ports system in many ways but where it differs it’s clearly superior. Also I love the cross-platform aspect and thus Raven is simply the better place for me to make home.

This year saw the introduction of Solaris/Illumos support that I tried out on OmniOS. Also Darwin support landed, upping the count of supported platforms to 5 already! Not too bad for a young project, huh? While Raven does work on all five platforms now it does so to varying degrees. But more on that later.

General activity

The Ravenports project consists of multiple Git repositories hosted on GitHub. The first one is Ravensource which most importantly holds the “raw” ports as they are written by the porters. It’s the most busy repo with over 5.200 commits since March 2017 (including almost 500 by me).

Then there’s the actual Ravenports repo that mostly contains the buildsheets which are compiled from Ravensource. It has over 1.400 commits right now.

Installing the xfce-single-core meta-package

Finally there’s the repo for the Ravenadm command-line tool. It’s approaching 900 commits since February 2017.

There’s still more to Raven like the Pkg package manager from FreeBSD (that was modified to add Zstd compression support) or libbsd4sol, a portability library which allows building code on Solaris that uses BSDisms (which was needed to add support for that platform to Raven). Most of the work on all repos was done by John alone.

With over 100 pull requests and more than 20 issues it’s clear now that there’s some interest in the project. Raven is still very small, though, with 6 people haveing contributed ports so far. After learning the basics and opening pull requests for half a year, I’ve been granted write-access to the source repository. Just recently I was able to push my 100th active port (there have been ports that became obsolete and were removed).

In general I’d say that there could of course be more people around and that the project would benefit from being able to provide more packages – though more than 3.200 is not bad at all! Also it’s good that there seems to be a growing user base which is even more important than having more porters join in. From my point of view, Raven is a healthy and fast-moving project. Still young, but doing well and heading in the right direction.

Major changes

There have been some pretty big changes that happened with Raven over time. Initially John started with a GCC6-based toolchain, only to switch to GCC7 when that was released. That was before my time with the project, but I witnessed the switch to GCC8.

Changing the toolchain certainly is a major interruption and most people are advised to just wait for the official repository to be re-rolled and then update. I had some bad luck in this regard – literally the day after I finally completed a working (and almost complete) set of basic packages for the FreeBSD_i386 platform, I faced the change to GCC8. Due to a lack of time I still haven’t repeated the switch on i386 (but I still plan to do it sometime).

The thunar file manager

Other changes that always have a huge impact (causing lots and lots of packages to be rebuilt) is adopting a new version (as well as dropping an old one) of the popular interpreter languages like Python, Perl and Ruby. Ravenports always supports two versions of Perl and Ruby and two versions of Python 3 (as well as 2.7 for now). So when Python 3.7 was released, 3.5 was removed and Perl 5.24 had to go when 5.28 was added.

Recently the former LLVM port that included everything regarding LLVM was split (LLVM, Clang, lld, openmp). Also now and then new statements are added to Ravenadm, so that old versions cannot work with a new release of the buildsheet repository (which is called “conspiracy”). But this is pretty easy to work around compared to the changes mentioned before.

So on the whole, Raven has proven that it can easily stand even big changes. For me this is essential to build faith in a project. And Raven is doing well in this regard.

Desktop-ready?

There are lots of people who will want to use Raven on servers. That’s totally fine of course. But for a project as ambitious as Ravenports, it’s necessary to provide a somewhat comfortable environment for the developers and the users alike. If it doesn’t manage to become a daily driver for people it cannot succeed.

For that reason I decided to work towards good desktop support for the little dev machine that I dedicated to my work on the project. When I started, X11 was already working and Openbox had freshly landed in the repos. So I had a simplistic environment to work with: Openbox + Xterm. However I could not even change my keyboard layout! Therefor I wrote a port for setxkbmap and eventually it was accepted as the first outside contribution to the project.

The Surf web browser

Next I did some work to get the FLTK toolkit and the EDE desktop in. Then I added my favorite terminal emulator, Sakura. This worked out pretty well and the biggest shortcoming at the end of 2017 was that there was no real graphical browser available. A lot has changed since then!

Desktop choices

Today you can choose between multiple window managers, both floating and tiling:

  • twm
  • cwm
  • openbox
  • fluxbox
  • xfwm4
  • pekwm
  • i3

And in case you prefer a real desktop environment, there are also several available:

  • Lumina (moderate, Qt-based)
  • Xfce4 (somewhat light-weight, GTK-based)
  • EDE (extremely frugal and minimalistic, FLTK-based)

Two graphical web browsers are available, Surf (which is deliberately simplistic and does not even support tabs) as well as an old version of Firefox (the last one that builds without Rust). This is certainly not perfect but much better than a year before.

Also other important programs are available, including LibreOffice! Last month the Apache webserver landed – which is a pretty complex port compared to many others.

Shortcomings

Are there packages you’ll miss? Most certainly. However there’s a wishlist now with ports that people would like to see created (please feel free to add more requests there). And that’s another good step ahead. Currently it’s almost 120 items long. Fortunately there’s been some success, too, and 26 requested ports have been created and taken of the list so far.

There are some future ports that will require lots of effort (hint: Help wanted!). The most important one that blocks some other important ports is the Rust compiler. There has been some work done on this but it’s not done, yet. Another real beast is TeX. This totally must be supported at some point. Current versions of Firefox and Chromium are often asked for. And somebody even requested Eclipse (which needs Java!). So there’s definitely more than enough work to do.

Using Raven on Linux works, but there are some flaws. Initially the Pkg package manager used to crash quite often. John traced that back to a bug in the version of SQlite that’s used internally by Pkg: The problem only struck on Linux and was fixed by using a newer version instead. While it’s much better now, there’s still the occasional problem with it.

While the packages from the repo work finde on Solaris 10u8 and above as well als Illumos, the exact version 10u8 is currently required to build packages. This is due to Solaris not being able to work with older system libraries in the build chroot. It would be great to haven an alternative ravensys-root for any Illumos distribution (OmniOS, SmartOS, Tribblix, …) available so that interested people without access to that specific closed-source Solaris version can develop Raven on that platform.

I don’t know how well Raven works on Darwin. Since I don’t have access to any macOS machines and PureDarwin is not really ready, yet, there’s currently no chance for me to test it. I intend to buy an older MacBook or something in the future, though, if I come across a fair offer and have some money available to spend on my hobby.

Some ports are not available on one platform or the other: Illumos mostly because they’d require patches to build and Linux often because it relies on additional libraries that have not yet been added to Raven. And then there’s a lot of packages that are mostly untested. All of these issues can be fixed, of course. All of those require a larger user-base, though. So it’s probably the best strategy to keep working on making Raven attractive to more users and address things when the right people show up.

What’s to come?

Currently Raven uses the primordial X11 input drivers (xf86-input-keyboard and xf86-input-mouse) on all platforms. In 2013 Linux pioneered support for generic input drivers by exposing the kernels “event devices”. Not too much later many Linux distributions adopted xf86-input-evdev. In 2014 there was a GSOC project to add evdev support for FreeBSD. Like many projects it came along a good part of the way but eventually was left unfinished. It was picked up and completed by a FreeBSD developer in 2016.

Xfce’s settings and applications menu

To use it, a special kernel had to be built so it would expose /dev/input device nodes. Then a sysctl had to be set – and eventually X11 had to be patched for emulated udev support… Why would anybody want to do all this just for different input drivers? Multi-touch support is just one valid reason. Another one is that having evdev-based input drivers is half the way to eventually support libinput, too. And that is one of the prerequisites for Wayland!

This month FreeBSD has finally enabled evdev support in the GENERIC kernel in both -CURRENT and 12-STABLE. That means the upcoming FreeBSD 12.0 will not support it out of the box, but most likely a future 12.1 will. Dragonfly BSD has also grown support for event devices and people are interested in working towards Wayland. I hope that we’ll be able to get xf86-input-evdev working with our X11 (on Dragonfly, FreeBSD and Linux) next year,

I’m taking a little break from Xfce now (but plan to port most of the remaining components later to make it a well-supported DE in Raven). There are a few things I have planned like adding Linux support for OpenVPN (it depends on some libraries and programs that are Linux only which are not yet in Raven). Also I intend to take a look at adding some more Qt5 components and write a few requested ports. And finally I want to write another post next year – a tutorial on using Ravenports and creating new ports.

So keep flying with us – it’s exciting times!

Ravenports: A modern, cross-platform package solution

This post is about Ravenports, a universal package system und building framework for *nix systems (DragonflyBSD, FreeBSD, Linux and Solaris at the time of this writing). It’s a relatively young project that begun in late February 2017 after a longer period of careful planning. The idea is to provide a unified, convenient experience in a cross-platform way while putting focus on performance, scalability and modern tooling.

What exactly is it and why should you care? If you’ve read my previous post, you know that I consider the old package systems lacking in several ways. For me Raven already does a great job at solving some problems existing with other systems – and it’s still far from tapping its full potential.

Rationale

A lot of people will think now: “We already have quite capable package systems. What’s the point in doing it again?” Yes, in many regards it’s “re-inventing the wheel”… And rightfully so! Most of the known package systems are pretty old now and while new features were of course added, this is sometimes problematic. There is a point where it’s an advantage to start fresh and incorporate modern ideas right from the start. Being able to benefit from the experience and knowledge gained by using the other systems for two decades when designing a new system is invaluable.

Ravenadm running on FreeBSD, OmniOS, Ubuntu Linux and DragonflyBSD

Ravenports was designed, implemented and is primarily maintained by a veteran in packaging software. John Marino at a time maintained literally thousands of ports for FreeBSD and DragonflyBSD. In addition to that, he wrote an alternative build tool called Synth. Aiming for higher portability, he modified Synth to work with Pkgsrc (which is available for many platforms) and also ported the modern Pkg package manager from FreeBSD to work with it.

In the end he had too many ideas about what could be improved in package building that would not fit into any existing project. Eventually Ravenports was born when he decided to give it a try and create a new framework with the powerful capabilities that he wanted to have and without the known weaknesses of the existing ones.

How does it compare to xyz?

It probably makes sense to get to know Ravenports by comparison to others. Let’s take a look at some of them first:

1) FreeBSD’s ports system is the oldest one such framework. It’s quite easy to use today, very flexible and since the introduction of Pkg (or “pkg-ng”) it also has a really nice package manager.
2) NetBSD adopted the ports system and developed it according to their own needs. It’s missing some of the newer features that FreeBSD added later but has gained support for an incredible amount of operating systems. Unfortunately it still uses the old pkg_* tools that really show their age now.
3) OpenBSD also adopted the early FreeBSD ports system. They took a different path and added other features. OpenBSD put the focus on avoiding users having to compile their own packages. To do so, they added so-called package flavors. This allows for building packages multiple times with different compile-time options set. Their package tools were re-written in Perl and do what they are meant to. But IMO they don’t compare well to a modern package manager.
4) Gentoo Linux with its portage system has taken flexibility to the extreme. It gives you fine-grained control over exactly how to build your software and really shines in that. The logical consequence is that, while it supports binary packages, this support is rudimentary in comparison.

EDE desktop, pekwm with Menda theme and brand-new LibreOffice

FreeBSD gained support for flavors in December 2017 and NetBSD did some work to support subpackages in a GSoC project in the same year. It’s hard to retrofit major new features into an existing framework, tough. When Ravenports started in the beginning of 2017, it already had those two features: Variant packages (Raven’s name for flavors) and subpackages. As a result they feel completely natural and fit well into the whole framework (which is why they are used excessively).

Ravenports knows ports options that can be set before building a package. Like with NetBSD or OpenBSD there’s generally fewer options available compared to FreeBSD. This is because Raven is more geared towards building binary packages than being a ports framework to build on the target machine (which would defeat the goal of always providing a clean building environment). For that reason the options mostly exist to support the variants for the packages. Compared to NetBSD’s Pkgsrc, Ravenports supports much fewer operating systems right now but has a much easier bootstrap process (binary!) for all supported platforms. It also offers a much superior package manager. When comparing against FreeBSD, OpenBSD and Gentoo, Ravenports is much more portable and supports multiple operating systems and – with the exception of FreeBSD – comes with a more modern package manager for binary packages.

Strong points

As Ravenports is not tied to a single operating system, it didn’t have to take into account specific needs that are for one OS only. In general there are no second-class citizens among the supported platforms. Also it was made to be agnostic of the package manager used. Right now it’s using Pkg only but other formats could be supported and thus binary packages be installed via pacman, rpm, dpkg, you-name-it.

Repology: Raven’s package freshness in percent (06/25/2018)

It allows for different versions of some software to be concurrently installed. If you e.g. want PHP 7.2 while some of your projects are stuck with 5.6 this is not a problem. It’s also possible to define a default version for databases like MySQL and Postgres as well as languages like Perl, Python and Ruby. Speaking of MySQL: Raven knows about Oracle MySQL, MariaDB, Percona and Galera. Only the first one is currently available (the ports for the others are missing) but the selection of which product to install is already present and the others can be easily added as needed.

If you build packages yourself you’ll notice that the whole tooling is fully integrated. Everything was planned right from the beginning to interact well and thus plays together just great. Also performance is something where Raven shines: Thanks to being programmed for high concurrency, operations like port scans are amazingly fast (if you know other frameworks).

Repology: Raven’s outdated package count (06/25/2018)

Raven follows a rolling-release model with extremely current package versions. In Repology, a fine tool for package maintainers and people interested in package statistics, Ravenports is the clear leader when it comes to freshness of the package repository: It rarely falls below 98% of freshness (while no other repo has managed to even reach 90% – and Repology lists almost 200 repositories!). If it does, it’s usually for less than a day until updates get pushed.

This is only possible because much of ports maintenance is properly automated. This saves a lot of work and allows for keeping the software version current without the need for dozens of maintainers. Custom port collections are supported if you have special needs like sticking to specific program versions. This way Raven can e.g. support legacy versions that should not be part of the main tree. It might also be interesting for companies that want to package their product for multiple platforms but need to keep the source closed. Ravenports supports private GitHub repositories for cases like this. All components of project itself are completely open-source, though, and are permissively licensed.

Also Raven is not the jealous kind of application. Packages are installed into /raven by default (you can choose to build your packages with a different prefix if you wish) and thus probably separate from the default system location for software. This makes it possible to use raven in addition to your operating system’s / distribution’s package manager instead of being forced to replace it.

Shortcomings

If you ask me about permanent problems with Raven: I don’t really see any. However there’s definitely a couple of things where it’s currently way behind other package systems. Considering how young the project is this is probably no wonder.

It’s a “needs more everything” situation. In fact it has the usual “chicken egg problem”: More available ports would be nice and potentially attract more users. With more users probably more people would become porters. And with more porters there’d surely be more ports available… But every new project faces problems like this and with resolve, dedication and perseverance as well as a fair amount of work, it’s possible to achieve the goal of making a project both useful and appealing enough for others to join in. Once that happens things get easier and easier.

KeePassXC, Geany and the EDE application menu

The Ravenports catalog has over 3,000 entries right now. It’s extremely hard to compare things like the package count, though. John provided an example: FreeBSD has 8 ports for each PostgreSQL version. With 5 supported versions that’s 40 ports. Ravenports has 5 ports with 8 subpackages each. In this case the package count is comparable, but not the port count. Taking flavors and multiversions into account, all repositories look much bigger than they actually are in case of available software. Also how to measure the quality of packages? What’s with ports that are used by less than a handful of people? What with those that are extremely outdated? Do you think they should count? It’s probably best to take a look and see if the software that you need is available. It is true though, that there’s of course still many important packages missing. IMO the most important one being Rust – which is not only needed for current versions of Firefox but increasingly important to build other software, too.

Also Linux support is not perfect, yet, and Solaris support even less so. On Solaris systems Raven is currently mostly binary-only because the Solaris kernel is unable to work with system libraries other than the ones matching exactly in version. Packages built on older releases of the OS work fine on newer ones, but for each OS release, a specific build environment would need to be created before building packages is possible. This is an issue that needs to be resolved in the future (I guess some help from the Illumos/Solaris community wouldn’t hurt). Also there are packages that don’t build on Solaris without patches which are not currently available. In case of important packages this leads to blockers since all other ports which depend on one such package also cannot be built: On FreeBSD there are 3,559 packages (including variants and metapackages) available from the repository at the present time. In the Solaris repo it’s only 2,851 packages. That’s certainly a nice start – but don’t expect to run a full-fledged desktop (or even X11 at all) there, yet!

In Linux land, distributions that come with glibc version 2.23 or newer work best. On distributions with older glibc versions (e.g. CentOS 7), software will not run as the standard C library is missing some required symbols. Raven will need to be bootstrapped again to support those distros. This is likely to happen before too long, but we’re not there, yet.

Current Firefox ESR version (+ sakura and pcmanfm in the panel)

MacOS (which might be supported soon), OpenBSD and NetBSD are not currently supported, nor is Linux with musl-libc or μclibc. Also currently Raven is amd64 only. ARM64 support is planned and i386 might or might not happen but are not available now.

Current status

At this time Raven is probably most interesting for people who love tech and enjoy tinkering on *nix systems as well as those who like the features and are ok with being early adopters. Yes, in general it’s ready for the latter. At least two people (including me) use Raven’s packages exclusively on one of their machines. I’d say it is ready as a daily driver (if you can live with the limited set of software available – or consider adding more ports). In fact I built a laptop that I use e.g. for on-call duty with it. Since that one is critical, it probably needs to be considered as “in production use”.

It’s possible to install various text mode applications with Raven, but X11 is also available. You can choose from multiple window managers or from at least two desktop environments (Lumina and the ultra-light EDE). Xfce4 is partially available (i.e. the panel is already ported). If you’re looking for web browsers, a current version of Firefox ESR (called “rustless-firefox”) can be installed as well as Surf, a simple webkit-based browser. The LibreOffice suite is available in its latest version, too. The same is true for the just released Perl 5.28 and Python 3.7.

Running Chocolate DooM and Chocolate Heretic

Oh, and if you’re into gaming… It’s not all just serious stuff. Yes, you can install and play DooM!

Conclusion

Ravenports is a fascinating project with lots and lots of possibilities. I wanted to get into porting with FreeBSD for quite a while but hesitated as I’m not a programmer. Then again I had been interested in package building for a long time and had played around with it on Arch Linux quite a bit. After my submissions to FreeBSD had been rotting in bug tracker for months (and still are after almost a year), I chose to give Raven a try in the meantime.

I was already familiar with Pkg and had used Synth before, too. Bootstrapping Raven’s pkg and then installing stuff was as easy as expected. The same was true for building the ports myself. Then I did quite a bit of reading and wrote my first port. It didn’t take more than 5 minutes after I opened my pull request on GitHub, before John responded – and the port was committed not much later. This was such a huge contrast that I decided to do more with Raven.

There was a learning curve, yes, but I received lots of help in getting started. I obviously liked the project enough to become a regular contributor and even got commit access to the ravensource repo later. Currently I’m maintaining just over 80 ports and I hope to write many more in the future. There have been some hard ports along the way (where I learned a lot about *nix), but lots of things are actually pretty easy once you get the hang of it.

Tongue-in-cheek: Make chaos or “make sense”!

If this post got you interested, just give it a try. Feel free to comment here and if you run into problems I’ll try to help. After this general overview of Raven the next post I plan to write will be on actually using it.