The history of *nix package management

Very few people will argue against the statement that Unix-like operating systems conquered the (professional) world due to a whole lot of strong points – one of which is package management. Whenever you take a look at another *nix OS or even just another Linux distro, one of the first things (if not the first!) is to get familiar with how package management works there. You want to be able to install and uninstall programs after all, right?

If you’re looking for another article on using jails on a custom-built OPNsense BSD router, please bear with me. We’re getting there. To make our jails useful we will use packages. And while you can safely expect any BSD or Linux user to understand that topic pretty well, products like OPNsense are also popular with people who are Windows users. So while this is not exactly a follow-up article on the BSD router series, I’m working towards it. Should you not care for how that package management stuff all came to be, just skip this post.

When there’s no package manager

There’s this myth that Slackware Linux has no package manager, which is not true. However Slackware’s package management lacks automatic dependency resolving. That’s a very different thing but probably the reason for the confusion. But what is package management and what is dependency resolving? We’ll get to that in a minute.

To be honest, it’s not very likely today to encounter a *nix system that doesn’t provide some form of package manager. If you have such a system at hand, you’re quite probably doing Linux from Scratch (a “distribution” meant to learn the nuts and bolts of Linux systems by building everything yourself) or have manually installed a Linux system and deliberately left out the package manager. Both are special cases. Well, or you have a fresh install of FreeBSD. But we’ll talk about FreeBSD’s modern package manager in detail in the next post.

Even Microsoft has included Pkgmgr.exe since Windows Vista. While it goes by the name of “package manager”, it turns pale when compared to *nix package managers. It is a command-line tool that allows to install and uninstall packages, yes. But those are limited to operating system fixes and components from Microsoft. Nice try, but what Redmond offered in late 2006 is vastly inferior to what the *nix world had more than 10 years earlier.

There’s the somewhat popular Chocolatey package manager for Windows and Microsoft said that they’d finally include a package manager called “one-get” (apt-get anyone?) with Windows 10 (or was it “nu-get” or something?). I haven’t read a lot about it on major tech sites, though, and thus have no idea if people are actually using it and if it’s worth to try out (I would, but I disagree with Microsoft’s EULA and thus I haven’t had a Windows PC in roughly 10 years).

But how on earth are you expected to work with a *nix system when you cannot install any packages?

Before package managers: Make magic

Unix begun its life as an OS by programmers for programmers. Want to use a program on your box that is not part of your OS? Go get the source, compile and link it and then copy the executable to /usr/local/whatever. In times where you would have just some 100 MB of storage in total (or even less), this probably worked well enough. You simply couldn’t go rampage and install unneeded software anyways, and sticking to the /usr/local scheme you separate optional stuff from the actual operating system.

More space became available however and software grew bigger and more complex. Unix got the ability to use libraries (“shared objects”), ELF executables, etc. To solve the task of building more complicated software easily, make was developed: A tool that read a Makefile which told it exactly what to do. Software begun shipping not just with the source code but also with Makefiles. Provided that all dependencies existed on the system, it was quite simple to build software again.

Compilation process (invoked by make)

Makefiles also provide a facility called “targets” which made a single file support multiple actions. In addition to a simple make statement that builds the program, it became common to add a target that allowed for make install to copy the program files into their assumed place in the filesystem. Doing an update meant building a newer version and simply overwriting the files in place.

Make can do a lot more, though. Faster recompiles by to looking at the generated file’s timestamp (and only rebuilding what has changed and needs to be rebuilt) and other features like this are not of particular interest for our topic. But they certainly helped with the quick adoption of make by most programmers. So the outcome for us is that we use Makefiles instead of compile scripts.

Dependency and portability trouble

Being able to rely on make to build (and install) software is much better than always having to invoke compiler, linker, etc. by hand. But that didn’t mean that you could just type “make” on your system and expect it to work! You had to read the readme file first (which is still a good idea, BTW) to find out which dependencies you had to install beforehand. If those were not available, the compilation process would fail. And there was more trouble: Different implementations of core functionality in various operating systems made it next to impossible for the programmers to make their software work on multiple Unices. Introduction of the POSIX standard helped quite a bit but still operating systems had differences to take into account.

Configure script running

Two of the answers to the dependency and portability problems were autoconf and metaconf (the latter is still used for building Perl where it originated). Autoconf is a tool used to generate configure scripts. Such a script is run first after extracting the source tarball to inspect your operating system. It will check if all the needed dependencies are present and if core OS functionality meets the expectations of the software that is going to be built. This is a very complex matter – but thanks to the people who invested that tremendous effort in building those tools, actually building fairly portable software became much, much easier!

How to get rid of software?

Back to make. So we’re now in the pleasant situation that it’s quite easy to build software (at least when you compare it to the dark days of the past). But what would you do if you want to get rid of some program that you installed previously? Your best bet might be to look closely at what make install did and remove all the files that it installed. For simple programs this is probably not that bad but for bigger software it becomes quite a pain.

Some programs also came with an uninstall target for make however, which would delete all installed files again. That’s quite nice, but there’s a problem: After building and installing a program you would probably delete the source code. And having to unpack the sources again to uninstall the software is quite some effort if you didn’t keep it around. Especially since you probably need the source for exactly the same version as newer versions might install more or other files, too!

This is the point where package management comes to the rescue.

Simple package management

So how does package management work? Well, let’s look at packages first. Imagine you just built version 1.0.2 of the program foo. You probably ran ./configure and then make. The compilation process succeeded and you could now issue make install to install the program on your system. The package building process is somewhat similar – the biggest difference is that the install destination was changed! Thanks to the modifications, make wouldn’t put the executable into /usr/local/bin, the manpages into /usr/local/man, etc. Instead make would then put the binaries e.g. into the directory /usr/obj/foo-1.0.2/usr/local/bin and the manpages into /usr/obj/foo-1.0.2/usr/local/man.

Installing tmux with installpkg (on Slackware)

Since this location is not in the system’s PATH, it’s not of much use on this machine. But we wanted to create a package and not just install the software, right? As a next step, the contents of /usr/obj/foo-1.0.2/ could be packaged up nicely into a tarball. Now if you distribute that tarball to other systems running the same OS version, you can simply untar the contents to / and achieve the same result as running make install after an unmodified build. The benefit is obvious: You don’t have to compile the program on each and every machine!

So far for primitive package usage. Advancing to actual package management, you would include a list of files and some metadata into the tarball. Then you wouldn’t extract packages by hand but leave that to the package manager. Why? Because it would not only extract all the needed files. It will also record the installation in its package database and keep the file list around in case it’s needed again.

Uninstalling tmux and extracting the package to look inside

Installing using a package manager means that you can query it for a list of installed packages on a system. This is much more convenient than ls /usr/local, especially if you want to know which version of some package is installed! And since the package manager keeps the list of files installed by a package around, it can also take care of a clean uninstall without leaving you wondering if you missed something when you deleted stuff manually. Oh, and it will be able to lend you a hand in upgrading software, too!

That’s about what Slackware’s package management does: It enables you to install, uninstall and update packages. Period.

Dependency tracking

But what about programs that require dependencies to run? If you install them from a package you never ran configure and thus might not have the dependency installed, right? Right. In that case the program won’t run. As simple as that. This is the time to ldd the program executable to get a list of all libraries it is dynamically linked against. Note which ones are missing on your system, find out which other packages provide them and install those, too.

Pacman (Arch Linux) handles dependencies automatically

If you know your way around this works ok. If not… Well, while there are a lot of libraries where you can guess from the name which packages they would likely belong to, there are others, too. Happy hunting! Got frustrated already? Keep saying to yourself that you’re learning fast the hard way. This might ease the pain. Or go and use a package management system that provides dependency handling!

Here’s an example: You want to install BASH on a *nix system that just provides the old bourne shell (/bin/sh). The package manager will look at the packaging information and see: BASH requires readline to be installed. Then the package manager will look at the package information for that package and find out: Readline requires ncurses to be present. Finally it will look at the ncurses package and nod: No further dependencies. It will then offer you to install ncurses, readline and BASH for you. Much easier, eh?

Xterm and all dependencies downloaded and installed (Arch Linux)

First package managers

A lot of people claim that the RedHat Package Manager (RPM) and Debian’s dpkg are examples of the earliest package managers. While both of them are so old that using them directly is in fact inconvenient enough to justify the existence of another program that allows to use them indirectly (yum/dnf and e.g. apt-get), this is not true.

PMS (short for “package management system”) is generally regarded to be the first (albeit primitive) package manager. Version 1.0 was ready in mid 1994 and used on the Bogus Linux distribution. With a few intermediate steps this lead to the first incarnation of RPM, Red Hat’s well-known package manager which first shipped with Red Hat Linux 2.0 in late 1995.

FreeBSD 1.0 (released in late 1993) already came with what is called the ports tree: A very convenient package building framework using make. It included version 0.5 of pkg_install, the pkg_* tools that would later become part of the OS! I’ll cover the ports tree in some detail in a later article because it’s still used to build packages on FreeBSD today.

Part of a Makefile (actually for a FreeBSD port)

Version 2.0-RELEASE (late 1994) shipped the pkg_* tools. They consisted of a set of tools like pkg_add to install a package, pkg_info to show installed packages, pkg_delete to delete packages and pkg_create to create packages.

FreeBSD’s pkg_add got support for using remote repositories in version 3.1-RELEASE (early 1999). But those tools were really showing their age when they were put to rest with 10.0-RELEASE (early 2014). A replacement has been developed in form of the much more modern solution initially called pkg-ng or simply pkg. Again that will be covered in another post (the next one actually).

With the ports tree FreeBSD undoubtedly had the most sophisticated package building framework of that time. Still it’s one of the most flexible ones and a bliss to work with compared to creating DEB or RPM packages… And since Bogus’s PMS was started at least a month after pkg_install, it’s even entirely possible that the first working package management tool was in fact another FreeBSD innovation.

Precomp (or: How to compress already compressed data?)

It’s a kind of strange feeling, but while half of the IT world seems to either already burn (or to tremble with fear), I can choose freely whatever topic I want to write about this month. I haven’t had a Windows box for almost a decade now and people who I work or keep in contact with, are also mostly *nix only. So this post is not about encryption or ransomware at all. It is about useful, respectable compression. Or more precise: The art of re-compressing already compressed data!

In January Precomp, a precompression utility, has been open-sourced! The first two sections tell a bit about how I became interested in this topic and in Precomp. Skip them if you don’t want to read that kind of stuff.

Compressing compressed data?

When I was young and new to PCs, I once tried to compress a ZIP archive with ACE (a lesser known archiver that once was comparable to the more popular RAR). I knew that ACE offered stronger compression and so I thought that this should make the file smaller. Just imagine my surprise when it turned out that I was wrong!

I guess that most of us have a story like that to tell, a story from our childhood when compression was nothing short of magic. Later when I begun to understand that even though it in fact does start with “m”, it’s not magic but math (a subject that I totally sucked at in school – but fortunately I grasped enough to get a rough idea on how compression works ;)). Now there was no surprise anymore: The compressed data is not well fit for any other general purpose compression method, even if it’s compressed with a weak algorithm.

How to work around that? Well, decompressing the ZIP file and creating a new ACE archive does the trick in the case mentioned above. Of course things are not always that straight forward. If they were, I wouldn’t really have much to write about right now and this post would be really, really short!

For whatever reason, compression continued to fascinate me and I loved compressing things to sizes as tiny as possible. It was fun to try out new experimental compression programs specialized on some specific types of files. I did that for years – until I had to stop due to a lack of time.

Games

Let’s fast forward some years from that failed compression experiment with ACE; I had replaced DOS 6.22 with Win95 which I had replaced with Win98 (SE) that I had replaced with WinME, … On some day I wanted to install Quake ]|[ Arena (yes, friends, I once was 1337 young enough to spell it like that!) on my main computer to get into it again for a LAN party next weekend. So I went looking for the darn CD. It took me a while but I finally found the CD case. I opened it up and… the CD itself was missing. Oh great! Since I didn’t feel like looking into all the other cases to find out into which I might have put it accidentally, I decided to just copy it off an older computer which had it already installed (ID were nice people. I don’t remember which version of Q3A it was, but there eventually was an official patch which also removed the CD check for the game so there was no need for a crack or anything).

Now, different versions of Windows didn’t always play together too well on the LAN and since my Quake installation was on a computer with an older Windows (and I didn’t have another cable at hand), I decided that I’d just burn it to CD. It turned out however, that the other machine didn’t have vanilla Q3A installed but the expansion set as well. Together it was obviously too big to fit on one CD. There would have been easy solutions: Leave out the resource files for the expansion, burn two CDs, put the hard drive into the new computer, … Sure, easy solutions are nice and all. But sometimes they are also boring! And when you’re young and have some free time, you don’t do boring stuff. So of course I opted for the more challenging solution: Get it all on one cd!

Quake 3’s resource containers go by the file extension of .pk3 and, more importantly, are in fact ZIP files without any compression. This meant that they could be compressed well because there was no ZIP compression getting in the way. But guess what: Even after applying the most extreme compression programs, the result simply would not fit onto one CD…

Bad luck, eh? Well, not really. Unpacking the container files was in fact the solution in this case. Not because of weak compression but because it enabled me to test each of the files it contained separately with all compressors and could group together all files that compressed best with one compression utility or another! I think that I was able to shrink it down almost as much as needed with just a couple of megs over the CD limit. There were blank CDs with 800 MB capacity as well, so it would have fit onto one CD – but I didn’t have one of those. So I replaced the ID video with an empty video file and I was set.

Since I liked doing these things I begun doing backups like that for a lot of my favorite games, ripping apart (and later rebuild) resource containers, convert between file formats, decompress whatever could be decompressed before applying stronger compression, etc.

How Precomp works

The more I got into free and open source things, the more I wondered if some of them wouldn’t benefit from better compression. A friend and former classmate of mine invented Precomp and I of course was among the first to make use of it and provide feedback. But what is Precomp?

Precomp is what the name says: A pre-compressor. It is not directly meant to reduce the size of files. On the contrary: It can make some files even bigger than the original input. But that’s a good thing really! How’s that? Well, it’s meant to prepare files for compression so that eventually these files can be compressed to a smaller size than the original file could – without losing data of course!

What Precomp does is look for streams in its input file that are compressed with a compression method known to Precomp. It then decompresses and recompresses them so that they can be compared. If they are identical, Precomp will write the decompressed stream (plus how to recompress it properly) to its output file.

While this sounds quite simple in theory, it is in fact a bit more complex. The reason for that lies in the flexibility of some compression algorithms. Have you ever zipped up a file? Then you know that there are a lot of parameters that you can provide which affects how the file will be compressed: “fast”, “normal”, “strong” or “maximum” compression? What about the dictionary size? A lot of things like that. So either combination of compression parameters will result in a valid zip stream that can be decompressed by any zip-compatible utility. Replacing such a stream with a compatible one is fairly easy. Reproducing the exact, bit for bit identical stream, is not.

To be truly lossless, Precomp uses trial and error on each stream. If it can figure out the combination of parameters that result in the original stream: Great! If not, that stream has to be left untouched.

What Precomp can do

Early versions of Precomp were only available on Windows but there have been Linux versions for quite a while as well. I also use it on FreeBSD without any problems. The .PCF files are platform-independent. You can restore the original file on Windows from a file precompressed on Linux or BSD and vice versa.

While Precomp originally was only a pre-compressor for zlib streams (which are used in a variety of file formats like ZIP, GZIP, PNG, PDF, …), it can do more things now. It can use bzip2 to compress its input file after precompression. It can losslessly compress some JPEG pictures to smaller sizes (thanks to an external library). And in the current development version there’s even support for compressing MP3 music files further (also using an external lib)!

Currently, Precomp relies on temporary files for all the extracted streams and thus puts heavy load on your hard drive (and is a bit slow due to that bottleneck). SSDs obviously perform better, but it totally makes sense to use a memdrive if you can spare some RAM for it. I’ve forked the project on Github and added an experimental shell script to assist with the creation of such a memdrive. It’s currently FreeBSD only (I’ve migrated all of my boxes to *BSD and currently have no Linux machine remaining but will set up one for cases like that some time in the future). Feel free to take a look at it if you’re into portable shell scripting and please do tell me if you have any suggestions!

Precomp is not at all at the limit of its possibilities. There are a lot of things that can be tweaked, optimized or added. If you feel like that could be a fun project – go ahead and play with it, it’s on Github. Or perhaps you have an idea what this could be useful for? Please help yourself and use it. It’s free software after all (Apache licensed).

Thea: The gain of giving away for free

This post is inspired by the game Thea: The Awakening. No, Eerie Linux has not mutated into a games blog. Yes, I will give a short description of the game. But what this post is really about is some thoughts about software development in the past, today and what could be a more open future.

Why Thea? Because the developers did something very uncommon: They decided to give the game away for free – if you’re a Linux user that is!

Thea: The Awakening

The game in question is a turn-based strategy game with a strong focus on survival. There’s a nice background story: The world had turned to darkness (playing the game you will discover why) and is haunted by creatures and spirits of the dark. Now the sun is rising again and the gods have returned but both are very weak and darkness will not give up without a fierce fight. Slavic mythology makes for a very nice and rather uncommon setting.

In case you want to give it a try, you can find a download link here. And yes, it is really completely free. You don’t need to buy the Windows version first or something.

I’ve successfully run the game on the Mint laptop that I share with my wife and can confirm that it works well. No luck on a 32-bit machine that I installed Arch on to give the 32-bit version of the game a try. It won’t start and the console messages give no clues why this may be. So if you’re still stuck with 32-bit only systems, you’re probably out of luck. 😉

The developers stated that they have not even tested the Linux version themselves! So what works and what doesn’t? Most things seem to work surprisingly well in fact. Sound, graphics, even the intro video. I’ve experienced graphical glitches with some white pixels appearing for a second (nope, no AMD video card – it’s Intel!). But this happens just rarely and is a fairly minor issue. Far more annoying is the fact that you cannot really use the keyboard: A key press works but the release event doesn’t… This is a known issue with the version of the Unity engine that Thea uses. It may or may not be addressed in a future release. You can however get the keys released by ALT-TABbing out of the game and back in. That way you can at least always access the menu.

You choose one of the gods when starting a game. I’ve played scenarios for multiple gods now. The main story (“Cosmic Tree”) gets pretty repetitive soon since it’s always the same. This is also true for a lot of the other quests. However the game has options to skip a lot of the text in case you already know it which certainly was a good idea. Some of the quests are different depending of which god you chose which keeps things interesting story-wise. Maps, resources, encounters, etc. are randomly generated for each game. This together with a challenging survival, plenty of combinations to try for crafting items and interesting gameplay, Thea might still cause a rather high motivation to replay the game often.

Software development models

I’d like to separate some development approaches here and sum them up by giving their model as I see it a name. These are no official models (I’m not a game developer) but an attempt to sum up the whole thing in one heading.

The shareware model

There was once a time when software was developed in a purely closed manner. It was developed internally and when it was ready, a release was done and advertised. The good thing was that games were often cut into “episodes” and the first one given away as shareware so people could try out the game for free and might decide to buy the full product.

The public relations model

Advertising grew bigger and bigger as well as more and more aggressive. Top titles games were often announced as development begun and some material was released along the development process to keep people hooked. This worked in some cases and failed in others (say Duke Nukem forever announced in 1996).

It was a reasonable move to try to build up an audience interested in a certain title early. The problem with that is mainly two things: You cannot keep people hooked for an arbitrary amount of time and such a continuing advertising campaign costs a whole lot of money way before you start earning anything from sales.

These problems lead to a new one, however. It puts very high pressure on the developers to meet deadlines to stay on schedule. And sometimes people in charge may even decide to release a half-baked product which almost always is a very bad idea… (what was the latest example? That Batman game perhaps?)

The community-aware model

It’s not a new insight that it is rather helpful for any title to have a large community. Some studios provide forums in an attempt to simplify building up of a community. And it’s also common knowledge today that feedback from that community is extremely valuable: Knowing your audience better helps a lot to provide the perfect product after all!

The most important point of this model is that interacting with the players is now bidirectional: There’s advertising targeting them but you certainly want to have (and honor) feedback provided by them. And it also makes sense to think about designing the game and/or providing the tools to easily modify the game and thus make it as easy as possible to create mods for the game. This can also be a huge plus when it leads to a bigger, more active and longer living community!

Independent of a single title, there is a possibility for a studio to get itself a good name by opening the source code for older games. This may require some cleaning up work first but some studios have also released code as-is (which can be rather terrible). But usually the community figures out what to do with it and before long the game is ported to new platforms, receives technical updates and enhancements. This has totally made some titles immortal: There are still new episodes, mods and total conversions for Wolfenstein being released. Yes, for a game from 1992 with extremely “poor graphics” (320×240, 8bit) by today’s standards! And there’s not one week without new maps for the mighty DooM (1993).

The community-supported model

There’s this interesting trend of “early access” games: Players are given the opportunity to playtest games before they are ready for release. People know they have to expect bugs but they can try out a game of their interest early and if they are very committed to it, they can report bugs as they encounter them.

This is a classical win-win situation: The developers get a broad testing done for free and the players can have a peak into the game early. Oh, and any form of interaction is of course always a good thing.

The community-backed model

That’s a rather new thing and basically means that some developers try to get their game crowd-funded. This can succeed and this can fail. There are examples for both cases. But while this is clearly a development model since it has a lot of impact on it, I’d say that it’s also more of a special case than a general model.

The future?

MuHa Games have made one clever step ahead with Thea as the gain of giving the title away for free on Linux is really considerable. How’s that? Well, if there was no Linux version, Linux people wouldn’t have bought the game, either. So giving it away is no actual loss: The number of people of the “hey, I would have bought it for Windows but why should I since I can play it for free on Linux!” kind are most likely extremely rare – if they exist at all.

No loss is fine, but where’s the actual gain? Well, there’s the “Just bought the Windows version. Besides: I don’t run Windows at all” type of guy. These people alone should suffice to cover the costs of the additional efforts to package a Linux release and upload it somewhere. But that’s not the main point at all: Can you say “Free advertising”? People talk about the game and people write about the game, many of which would not have done it if it had just been an ordinary game! Now with the free Linux release the game, MuHa managed to make it stand out (and that is not too easy today).

For these reasons giving it away proves to be a very sensible PR action! I do not mind if that was intended or not. That doesn’t change the facts.

Community-assisted model?

So what could the future hold? I can imagine that making the community engage even more would be a big benefit. From a studio’s perspective, fans do unpaid work because they love the product. And from the fan’s perspective it’s just cool to be part of one of your favorite games and help improve it.

What could this look like? My vision is to sort of blend closed source development with what we learned from open source development. It’s cool that people playtesting a game can report bugs via forum or email. But when will the first project set up a public bugtracker along with a tutorial on how to use that for bug reports and maybe (sensible) feature requests?

Then: What about translation? Open source achieved made very, very good results using translation frameworks like Transifex. Now Thea is only available in English. My native language is German and I would not have minded at all to dedicate some time translating a few strings (I got a nice game for free after all!). There’s a lot of potential in this.

And along that it would totally make sense to avoid using proprietary containers for files. I did not bother to try to extract text out of whatever format it is that MuHa uses for Thea. In 1999 ID Software did a clever thing for Quake III Arena: They used container files called “.pk3” – which were simply renamed, uncompressed Zip files. The benefit is obvious: Everybody can extract the resources, modify them and put things back together. Great! I noticed a lot of spelling mistakes in Thea. If I had had access to the game text you’d have received a series of patches from me (and by applying they you’d instantly see which ones are still valid and fixing mistakes). Wouldn’t that be a great way to improve the game?

Licensed Open Source model?

Can open source work for a commercial game? Well, why not? Open source alone does mean just that: The source is open. It does not say under which license and it does not say that it’s free. Now I generally support as much freedom as possible – but that last word there is important. A more open development is a nice improvement IMO. There’s no reason to demand more than that.

In this model the customers pay for the game data without which you obviously cannot play the games but the program source is open (or perhaps semi-open where it is included with the copy of the game you get when you buy it and you’re free to distribute a series of patches but not the source itself). I’m pretty sure that this can work. One potential problem here may be deadlines. Often the code in commercial games must be horrible – not because the programmers suck but because unrealistic deadlines blow. A lot of studios may hesitate to open up their code just for that very reason…

Addressing the problem could however also be easy: You sell games in early access? Buyers get the code and know that it’s early and may not be in perfect shape (and can actually help improving it). Again both sides win: The studio gets code review and maybe some patches plus some people may even attempt to port the game to platforms unsupported by the studio. The players get better games they can help to improve, take modding to the next level and even a chance to see what coding is like and get yourself some reference work if you intent to work in that industry!

There’s one other issue, though. In many cases studios will want to hide some things from competitors. That may be old (and at some point hopefully obsolete) thinking but we have to accept it as a present fact. So what about this? Well, those things could be put into libraries… It’s far better to have the program code open and make it use closed libraries than having nothing open at all!

Time for change

Who’s stepping forward making the next step in game development? I’m really curious if something in the direction of what I wrote here happens any time in the future. For each step there’s good press to catch for free again, you know? 😉 Perhaps some small studio dares to make the move.

Update: I wrote this in a hurry on 11/30 to rush out my November post. And then I once again forgot to make it public. But now it is…

An interview with the Nanolinux developer

2014 is nearly over and for the last post this year I have something special for you again. Last year I posted an interview with the EDE developer and I thought that another interview would conclude this year of blogging quite fine.

In the previous post I reviewed Nanolinux (and two years ago XFDOS). Since I was in mail contact with the author about another project as well, it suggested itself that I’d ask him if he’d agree to give me an interview. He did!

So here’s an interview with Georg Potthast (known for a variety of projects: DOSUSB, Nanolinux and Netrider – to just name some of them) about his projects, the FLTK toolkit, DOS and developing Open Source software in general. Enjoy!

Interview with Georg Potthast

This interview was conducted via email.

Please introduce yourself first: How old are you and where are you from?

I am 61 years old and live in Ahlen, Germany. This is about 30 minutes drive from Dortmund where they used to brew beer and where the BVB Dortmund soccer team is currently struggling.

Do you have any hobbies which have nothing to do with the IT sector?

Not really. I did some Genealogy, which has to do a lot with IT these days. But now I have several IT projects I am working on.

DOS

You’re involved in the FreeDOS community and have put a lot of effort into XFDOS. A lot of people shake their heads and mumble something like “It’s 2014 now and not 1994…” – you know the score. What is your motivation to keep DOS alive?

I have been using DOS for a long time and wish it would not go away completely. So I developed these DOS applications, hoping to get more people to use DOS. But I have to agree that I have not been successful with that.

Potential software developers find only very few users for their applications which is demotivating. Also there is simply no hardware available today that is limited so much that you better use DOS on it. Everything is 32/64 bit, has at least 4 GB of memory and terabytes of disk space. And even the desktop PC market is suffering from people moving to tablets and smartphones.

People are still buying my DOSUSB driver frequently. They are using it mostly for embedded applications which shall not be ported to a different operating system for one reason or another.

Do you have any current plans regarding DOS?

I usually port my FLTK applications to DOS if it is not too much effort to do so. So they are available for Linux, Windows and DOS. Such as my FlDev IDE (Link here).

Recently I made a Qemu/FreeDOS bundle named DOS4WIN64 (Link here) that you can run as an application on any Windows 7/8 machine. This includes XFDOS. I see this as a path to run 16bit applications on 64bit Windows.

How complicated and time consuming is porting FLTK applications from Linux to DOS or vice versa?

It depends on the size and the dependencies on external libraries. I usually run ./configure on Linux and then copy the makefile to DOS where I replace-lXlib with -lNXlib plus -lnano-X. Then, provided the required external libraries could be downloaded from the djgpp site, it will compile if the makefile is not too complicated (recursive). Sometimes I also compile needed libraries for DOS which is usually not difficult if they have a command line interface.

You then have to test if all the features of the application work on DOS and make some adjustments here and there. Often you can use the Windows branch if available for the path definitions.

Porting DOS applications to Linux can be more complicated than vice versa.

Linux

For how long have you been using Linux?

I have been using Linux on and off. I began using SCO-Unix. However, I did not like setting things up with configuration files (case sensitive) scattered over many directories. It took me over a week to get serial communications to work to connect a modem. When I asked Linux developers for help they recommended to recompile the kernel first – which means they did not know how to do that either. So I returned to DOS at that time. But I have been using Linux a lot for several years now.

What is your distribution of choice and why?

I mainly use SUSE but I think Ubuntu may work just as well. This may sound dull but you do not have to spend time on adding drivers to the operating system or porting the libraries you need. The mainstream Linux distributions are well tested and documented and you do not have to spend the time to tailor the distro to your needs. They do just much more than you need so you are all set to start right away.

My own distro, Nanolinux, is a specialized distro which is meant to show how small a working Linux distro can be. It can be used on a flash disk, as an embedded system, a download on demand system or to quickly switch to Linux from within Windows.

However, if you have a 2 Terabyte hard disk available I would not use Nanolinux as the main distribution.

FLTK

Which programming languages do you prefer?

I like Assembler. To be able to use X11 and FLTK I learned C and C++ which I currently use. I have not done any assembler in a while though.

You seem to like the idea of minimalism. Do you do use those minimalist applications on a daily base or are they more of a nice hobby?

Having a DOS and assembler background I always try not use more disk space than necessary. Programming is just my hobby.

Many of your projects use the FLTK toolkit. Why did you choose this one and not e.g. FOX?

I had ported Nano-X to DOS to provide an Xlib alternative for DOS developers. In addition I ported FLTK to DOS as well since FLTK can be used on the basis of Nano-X. So I am now used to FLTK.

Compared to the more common toolkits, FLTK suffers from a lack of applications. Which three FLTK applications that don’t exist (yet) do you miss the most?

I think FLTK is a GUI toolkit for developers, so it is not so important what applications are available based on FLTK.

If you look at my Nanolinux – given I add the NetRider browser and my FlMail program to the distro – it comes with all the main office applications done in FLTK. However, the quality of these applications is not as good as Libre Office, Firefox or Gimp. I do not expect anyone to write Libre Office with a FLTK GUI.

When you awake at night, a strange light surrounds you. The good FOSS fairy floats in the air before you! She can do absolutely everything FOSS related; whether it’s FLTK 3 being completed and released this year, a packaging standard that all Linux distros agree on or something a bit less unlikely. 😉 She grants you three wishes!

As with FLTK 3 I wish it would change its name and the development would concentrate on FLTK 1.3.

Regarding the floating fairy I would wish the internet would be used by nice and friendly people only. Currently I see it endangered by massive spam, viruses, criminals and even cyber war as North Korea apparently did regarding the movie the ruling dictator wanted to stop being shown.

Back to serious. What do you think: Why is FLTK such a little known toolkit? And what could be done about that?

I do not think it is little known, just most people use GTK and so this is the “market leader”. If you work in a professional team this will usually decide to go for GTK since most members will be familiar with that.

What could be done about that? If KDE and Gnome would be based on FLTK I think the situation will change.

From your perspective of a developer: What do you miss about FLTK that the toolkit really should provide?

Frankly speaking, as a DOS developer the alternative would be to write your own GUI. And FLTK provides more features than you could ever develop on your own.

What I do not like is the lack of support for third party schemes. Dimitrj, a Russian FLTK developer who frequently posts as “kdiman” on the FLTK forums, created a very nice Oxy scheme. But it is not added to FLTK since the developers do not have the time to test all the changes he made to make FLTK look that good.

What do you think about the unfortunate FLTK 2 and the direction of FLTK 3?

I think these branches have been very unfortunate for FLTK. Many developers expected FLTK 2 to supersede FLTK 1.1 and waited for FLTK 2 to finish before developing an FLTK application. But FLTK 2 never got into a state where it could replace FLTK 1.1. Now the same seems to happen with FLTK 3.

So they should have named FLTK2/3 the XYZ-Toolkit and not FLTK 2 to avoid stopping people to choose FLTK 1.1.

Currently there is no development on FLTK 2/3 that I am aware of and I think the developers should concentrate on one version only. FLTK 1.3 works very well and does all that you need as a software developer as far as I can say.

Somebody with a bit of programming experience and some free time would like to get into FLTK. Any tips for him/her?

I wrote a tutorial which should allow even beginners in C++ programming to use FLTK successfully (Link here).

Nanolinux

You’ve written quite a number of such applications yourself. Which of your projects is the most popular one (in terms of downloads or feedback)?

This is the Nanolinux distro. It has been downloaded 30.000 times this year.

NanoLinux… Can you describe what it is?

Let me cite Distrowatch, I cannot describe it better: Nanolinux is an open-source, free and very lightweight Linux distribution that requires only 14 MB of disk space. It includes tiny versions of the most common desktop applications and several games. It is based on the “MicroCore” edition of
the Tiny Core Linux distribution. Nanolinux uses BusyBox, Nano-X instead of X.Org, FLTK 1.3.x as the default GUI toolkit, and the super-lightweight SLWM window manager. The included applications are mainly based on FLTK.

After compiling the XFDOS distro I thought I would gain more users if I would port it to Linux. The size makes Nanolinux quite different from the others and I got a lot of downloads and reviews for it.

The project is based on TinyCore which makes use of FLTK itself. Is that the reason you chose this distro?

TinyCore was done by the former main developer of Damn Small Linux. So he had a lot of experience and did set up a very stable distro. Since I wanted to make a very small distro this was a good choice to use as a base. And I did not have to start from scratch and test that part of the distro forever.

NanoLinux uses an alternative windowing system. What can you tell us about the differences between NanoX and Xorg’s X11?

Nano-X is simply a tiny Xlib compatible library which has been used in a number of embedded Linux projects. Development started about 15 years ago as far as I recall. At that time many Linux application developers used X11 directly and therefore were willing to use an alternative like nano-X for their projects.

Since nano-X is not fully compatible to X11, a wrapper called NXlib was developed, which provides this compatibility and allows to base FLTK and other X11 applications on nano-X without code change. The compatibility is not 100% of cause, it is sufficient for FLTK and many X11 applications.

Since nano-X supported DOS in the early days I took this library and ported the current version to DOS again.

Netrider

The project you are working on currently is NetRider, a browser based on webkit and FLTK. Please tell us how you came up with the idea for it.

Over the years I looked at other browser applications and thought how I could build my own browser, just out of interest. Finally Laura, another developer from the US, and I discussed it together. She came up with additional ideas and thoughts. That made me have a go at WebKit with FLTK.

What are your aims for NetRider?

I wanted to add a better browser to my Nanolinux distro replacing the Dillo browser. Also, as a FLTK user I wanted to provide a FLTK GUI for the WebKit package as an alternative to GTK and Qt.

There’s also the project Fifth which has quite similar aims at first sight. Why don’t you work together?

Lauri, the author of Fifth, and I started out about the same time with our FLTK browser projects, not knowing of each other’s plans. Now our projects run in parallel. Even though we both use FLTK, the projects are quite different.

We have not discussed working together yet and our objectives are different. He wants to write an Opera compatible browser and competes with the Otter browser while I am satisfied to come up with something better than Dillo.

I did not ask Lauri whether he thinks we should combine the projects. I am also not sure if this would help us both because we implemented different WebKit APIs for our browsers so we would have to make a WebKit library featuring two APIs. This could be done though. Also he is not interested in
supporting Windows which Laura and I want to support.

Would you say that NetRider is your biggest project so far? And what plans do you have for it?

Setting up Nanolinux and developing/porting all the applications for it was a big project too, and I plan to make a new release beginning of next year.

As with NetRider it depends if people like to use it or are interested to develop for / port it. Depending on the feedback I will make my plans. Recently I added some of the observations I got from beta testers, did support for additional languages, initial printing support etc.

The last one is yours: Which question would you have liked me to ask in addition to those and what is the answer to it?

I think you already asked more questions than I would have been able to come
up with. Thank you for the interesting questions.

Thanks a lot Georg, for answering these questions! Best wishes for your current and future projects!

What’s next?

I have a few things in mind… But I don’t know yet which one I’ll write about next. A happy new year to all my readers!

The concepts of complexity and simplicity

Life in general is a very complex thing. Society is a complex matter, too. Also the IT world is a complex one. And so are many of today’s programs – for the good or the bad.

In many fields complexity is a necessity. You cannot build a simple microprocessor that satisfies today’s needs. And there is no way to have a very simple kernel that can do everything that we need. I agree to that and I do not want to condemn complexity as a whole. But – and I cannot stress that enough – while more and more sophisticated programs are being developed, projects have the tendency to become overly complex. And this is what I criticize.

A bit of history

Most of my readers are probably happy users of some Unix-like operating system. Some may live long enough to have witnessed how these systems changed over time. Many of us younger ones did not and so we only know what we have read about these times (or probably not even that).

Thinking about the heritage of Unix, another OS called Multix comes to one’s mind. This system was jointly developed by AT&T, GE and the MIT. It was a sophisticated operating system which had many remarkable or even truly revolutionary features for its time. A lot of effort and money was put into it. High expectations were put on Multics. And then eventually – it failed.

AT&T had pulled out of the project when they realized that it was rather slow and overly complex. They learned from it and attempted to create a system which followed the opposite approach: Aim for simplicity. This system lead to an incredible success: Unix.

So it is important to know that enthusiasm for technology and the urge to develop more and more complex programs is not a new phenomena at all. In fact I’d claim that it is the logical consequence of how man thinks. While all things begin with relatively simple forms, complexity as a concept does not follow after the concept of simplicity. On the contrary: Simplicity is the lesson learned after realizing the downsides of complexity.

Universalism and particularism

Some people seem to be fascinated with the idea to have one tool that should do nearly everything. Let’s assume we have that tool available today. The result will be an extremely complex application which has an overwhelming number of features. There will hardly be any single person who will know all these features (let alone bring all of them to use).

Now each feature you don’t use wastes space on your drive. While this is true, it is certainly the smallest problem when you’re not working in the embedded field. A bigger one is that it will surely be of low quality: While it can do a hell of a lot of things, it is very unlikely that all of its features will be comprehensive. The program is likely to be rather slow because optimizing a very complex program is extremely difficult. The worst thing however is that it is bound to contain a high amount of bugs, too!

It is a well-known fact that program code where functions are longer than the maximum lines that fit on the screen, contain far more bugs. For some reason a lot of programmers seem not interested in writing good code but either just want to get something done or aim at too ambitious goals which make the project overly complex.

On the other hand there are projects which specialize in a single, narrow field. If you suggest a new feature it may very well happen that it will be rejected. The people who work on this project do not care for stuff just because that’s currently ultra-hip. Instead they often refer to features which are not really needed as unnecessary bloat. These programs cannot do a lot of things by themselves but excel at what they can do.

Following the later idea is the Unix way of doing things. The true power comes from the combination of specialized tools which can yield mind-blowing results when used by an experienced user.

Featuritis?

There are quite a few programs which suffer from a strange illness which could be called “featuritis”. It often makes the host look handsome and appealing for many people. This illness is usually not deadly and often invisible for quite some time. But it does bear a very destructive aspect, too…

Two of the programs recently found infected are OpenSSL and BASH. The former kept so much legacy code in the project and even re-implemented things done better by others that it was impossible to have a good overview of the whole project code. The later implements a lot of features which are hardly ever used by anybody and also uses some functions of its own which are arguably wasted code since there are better alternatives out there.

Both projects succeeded in being widely distributed but read by few and understood by even fewer. And those few didn’t look at all the obscure parts of those unclear and confusing code. This is why severe bugs could exist for a very long time before anybody ever noticed.

Probably the most important project where I diagnose a particularly intense form of featuritis is Systemd. It acts like an init system but absorbed the functionality of so many other programs by now that I’m getting dizzy thinking of it. Worse: A lot of people who have looked at it more than just a bit claim that it is badly designed and the code is rather unclean. Even worse: The developers of Systemd have had a conflict with Linus Torvalds because they broke things with their code and even refused to fix it insisting that it was not their problem! And the true tragedy is that it has spread to a great many Linux distros. Once a really bad bug is found concerning Systemd, this will probably take suffering for admins and users to a whole new level.

An exit strategy for the brave

My respect for the OpenBSD guys continues to grow the more I read about it. They claim to have a very secure OS and from what they do I can only say that they mean it. The LibreSSL fork or the SystemBSD project are just two examples that show how dead serious they are. A lot of people seem to ridicule them because there are not too many OpenBSD users out there when compared to Linux. That’s true of course. Their OS may also not be very familiar from a Linux user’s point of view and the OpenBSD guys may not be too friendly towards newbies. But they are nice enough to make their projects portable so that everybody can profit from them!

And in case you want to stick with Linux, there’s a great source for this platform as well. The guys over at suckless aim at creating programs “that suck less”. Go ahead and read a bit – especially the sucks and rocks pages! On the first one you’ll flabbergasted at how bad the situation really is with a lot of programs. Yes, they are fundamentally broken – and their developers don’t care about that. Code correctness doesn’t pay of if you just want to target the masses. But if you want to do things right it does.

Are there really people out there who care? You bet there are. Think about this topic again and try out a few alternatives. You might well find a real gem here and there – if you are able to look over some of the shortcomings compared to the well-known, featureful and bloated defaults.

Software licenses (pt. 1): A general introduction

This is obviously not about the EERIE distro or Arch:E5. The reason for that is simply that I didn’t succeed in getting everything working. And to be honest, I didn’t have much time to attempt it in the first place. My second child was born this month and I guess everybody would agree that family comes first. So here’s the first post of a series that I had in mind since well before Christmas. Time just goes by so damn fast!

I’ve been thinking about software licenses a bit and decided to write about it. It’s a rather special topic for sure. Many programmers like dealing with licenses just as much as they like to write documentation: Not at all. For that reason a lot of people who support open source software decide to take the easy way and simply GPL their code.

However licensing is a very important thing and should be taken seriously. This article is meant to give a quick introduction – while avoiding the major problem of the whole license issue: Being boring for most people!

What is a license?

Ever read a typical Microsoft EULA (“End User License Agreement”)? No? Well, it’s not just you. Most people haven’t. Still you probably should. Or at least read a bit of it. Even if you didn’t read the license, you’ve agreed to it if you’re using Microsoft software. And that means that you are bound by its content – no matter if you’re in fact ignorant about that.

But what is a license? In fact you can think of a license which comes with a program as a blueprint. It is a draft of what the author proposes to you. Simply put: If you accept it, it will become a valid legal treaty. Yes – accepting a license isn’t some neglectable action. In doing it, you’re signing a treaty which legally binds you.

In general licenses contain various items which permit and prohibit certain actions. For example you may be granted the right to install one copy of a program on one computer and use it. You’re usually forbidden to analyze the program by means of reverse-engeneering. However there may be additional requirements being imposed on you. Probably you have to pay a monthly fee to continue using some service or you’re required to supply a valid address and keep it up to date.

Once you understood that by accepting a license you’re signing a treaty, you might no longer do so carelessly. After all you’re legally accountable if you knowingly or unknowingly violate it.

There has been a steady development in terms of proprietary licenses over the past decade. Many of them are getting stricter and more intrusive all the time. There are probably some quite nasty passage in the next license you’re going to “agree” to. So you should probably care about it.

(Why) do licenses matter?

Software – like any other artificial thing – doesn’t emerge out of nowhere. It has one or more author(s) who wrote the program and dedicated time and effort to that project. Now it is absolutely comprehensible that the author gets to decide what he or she would like to with it. The creator of some software project may decide to keep it private, try to sell it or give it away for free. And of course it’s all the author’s decision whether to open-source the project or to keep it close.

Depending on the nature of the program either path may seem like a good choice. The really important thing however is that each possible decision is perfectly legitimate. Who creates something decides. So if a coder makes the decision to give the source to his program away to the public – then this project is open-source and we can do all the good things with it, right? Wrong. Unfortunately.

Our coder may give the source of his or her program away for free but still remains the sole copyright holder. If the code is made available to you that implies that you may take a look at it. Certainly a nice move of the author. But other than that you’re not actually allowed to do anything with it. It is the intellectual property of another person. So without permission you may not redistribute it, are not allowed to modify it or in fact to actually put it to some use! Yes: In each of these cases you’d have to ask for permission first.

Let’s assume you cannot reach the author since he or she got a new email address and didn’t update the project page. Or the author simply doesn’t have the desire (or time) to answer mail asking for permissions. Well, if you’re a coder yourself, the source code may help you to get an idea on how to solve certain problems. But that’s pretty much it. If you need some functionality you’ll have to repeat the work already done and reimplement it because you’re not allowed to reuse the code despite it being publically available.

So for these reasons the answer to the question in the headline can only be: Yes. Licensing does matter!

What’s next?

I currently have three things in the making: 1) The basic repo of a musl-based Arch-like Linux distro 2) Quite some i686 packages for ArchBSD 3) An article about “Optimistic and pessimistic licenses”. And 4) I have no idea how much time I can spend before my screen in the coming weeks. 😉

Toolkits – an introduction

The topic of this post is GUI (Graphical User Interface) toolkits. As I wrote before, the EERIE blog is for both Linux veterans and novice users. This entry is meant for the later. If you already have more than a rough idea about what toolkits are, save yourself some time and skip it.

Why to give thought about toolkits?

The importance of the decision which desktop environment to use is pretty obvious. In case of GUI toolkits this is less self-explanatory. However deciding on the right TK is a crucial issue when planing a new distribution. But what are toolkits and what do they do?

While this is not too complicated a subject, it can also be a bit confusing. Especially since there are several commonly used terms which mean the same thing and also some often used in a semi-wrong way that can easily lead to misunderstandings. But let’s shed some light on this topic!

Low-level and high-level programming

With any pc it’s the CPU that can execute a number of instructions. While the set of special instructions grew bigger and bigger (think of MMX, SSE, etc.) over time, it’s generally a big hassle to create more complex programs entirely by doing low-level programming.
Here’s one example in Assembler:

MOV AH,BX

This line translates more or less to “put the value of register AH into register BX“. It’s not hard to imagine that coding like that is complicated, tedious and not very intuitive at all (the programmer has to keep an overview of what is in which register in our example).

For this very reason high-level programming languages exist: it’s far easier to do complex tasks in a more abstract way that is closer to what the human brain works like. It’s not hard at all to tell what the following line does:

PRINT "Hello world!"

Yes, this code fit for the language Basic just prints the well-known message to the screen. Basic is for the main part an “interpreter language”. This means that a set of commands is “translated” into machine code by the interpreter. While Basic provides a fine example here, it’s a little too simple to be fit for serious programming today. There are many high-level languages available, probably the most common being C and C++.

Source code, compilers & libraries

If a programmer wants to create a new program, he or she writes the source code (all the commands which produce the program in the end) for it in his or her prefered language. Afterwards the source code gets compiled (and linked) and the result is an executable program file.

While all high-level languages offer a certain set of commands, it would still be very cumbersome if all programmers had to stick to just these. It absolutely makes sense not to re-invent the wheel each and every time but to simply use some advance routines. Those are often provided by various libraries which are not by themselves programs for the end-user but in fact act like collections of functions which can be called by other programs.

Printing some string on the screen is a typical example. Almost every program needs to have this ability. Every programmer could write his or her own routines or just use an external one to do it. In the later case the programmer also has to decide how to ensure that his program can relay on the external resources. There are two ways to do this!

Static and dynamic linking

It’s possible to append the library to your program executable – this is called static linking. This is obviously the safest way. The drawback however is that this increases the size of the executable. In case of bigger projects which use a lot of libraries this is not a good idea at all. In addition this may be a wast of disk space if you have several programs on your computer which use the same library and each one has it statically linked!

The other possibility is to use dynamic linking. In this case the external resource stays external: the program relays on a library that needs to be installed on your computer. These libraries are called *.so (shared object) on Linux systems and *.dll (dynamic link library) on Windows. To use these is much more efficient in terms of space. However it can be a nuisance if a program depends on a lot of libraries and it can also be a problem if one program relays on an old version of a library and doesn’t work with a newer one while another program needs the latest one…

GUI toolkits

A GUI toolkit, also known as widget toolkit or sometimes widget library is a special kind of library. Their purpose is to provide a set of “widgets” (like a button for example) which other programs can use. This ensures that any e.g. a button drawn by any application using the same TK follows the same style (i.e. looks alike). Many toolkits do not just support one platform but are cross-platform TKs. This means that you could write a program based on sometoolkit which could then run e.g. on Linux, MacOS and Windows without having to care about how the particular system manages drawing graphics on the screen!

However there are several toolkits available – some a bit lighter, some a lot heavier. Of course they work differently and offer a different function volume. But those things will be discussed at a later time.

What’s next?

The next post will be about something really neat that I discovered recently. I won’t spoil it yet, but it’s something that probably nobody would ever expect. I just have to write about it! After that we’re going to examine the toolkits a little closer.