Eerie Linux: 5 years of bloggin’!


The Eerie Linux blog silently turned 5 years just last month. I thought a while about what kind of anniversary post I should write to celebrate the fifth birthday. I was even thinking of closing the blog on that day or at least announce that I would no longer be able to write posts regularly. I decided against it. While I don’t make any promises, I will try to keep the blog up for now.

The June marathon

In the end I decided not just to hold back that birthday post (this one) but do something special instead: Write a full article every five days! It was a lot of work, but June 2017 saw 6 posts each with over 1,600 words on average with one just falling short on 2,000. I put a lot of detail into those posts and also included quite some pictures.

It has been a fun experience but also an exhausting one. I have always been pressed for time and even though I tried to create as much material on weekends if the targeted date was during the week. Still I almost never managed to complete a whole article before the day it was due and often had to finish it in the late hours of evening after work. But now it’s done and I’m happy about that! 😉

5 years of blogging

A lot has happened in the last 5 years. When I started the blog in June 2012, I had quite some time on my hands but I wasn’t sure if I would always find enough topics to write about. This has changed completely. Free time is pretty scarce these days but there’s just so much going on in technology and related areas that I have a very, very long list of things that I’d like to write about – and that list grows faster than I can write and publish articles.

I’ve also moved houses three times over these years – and still haven’t missed a single month completely. Each and every month has had at least one new article and I’m a bit proud of that because a lot of times it really hasn’t been easy.

Since 2013 every year I get most page hits from the US with Germany being second. Ranks 3+ vary.

2012

After thinking about starting a blog for over a year, in 2012 I actually started it. I had been using SuSE and Ubuntu Linux on the desktop for a while and wanted to know more about the operating system. And I figured that it would make sense to pick an ambitious but realistic project and write about it as the journey continued.

In my first half-year I wrote 24 posts introducing myself, finding a suitable distro (looking info Gentoo first but then settling for Arch), thoughts on graphical toolkits and so on. The most important articles were part of a series on installing and comparing 20 Linux desktop environments.

The 6 month of 2012 saw just over 1,000 page views and I even got my first “likes” and comments. However I had no idea if I was doing good for a blog of that kind. Considering that it was public and that the whole world could potentially visit the blog, it seemed pretty low. Especially if you consider the many hours that went into the posts. “There must be thousands of Linux blogs out there and who should read them all?”, I thought. But I went on doing what I was doing because of my own interest in Linux topics. And I also continued to blog about it. If somebody would read and enjoy it: Execllent. If not it had at least made me write an English text which is quite valuable for the non-native speaker.

2013

In retrospective, 2013 was an interesting year. I got the most comments and “likes” that I ever got in a single year. And page hits increased to just over 6,600! You can imagine that I was extremely happy that there actually proved to be some interest in what I was doing. I already had less time now and managed to write 22 posts in the whole year instead of 24 in 6 just months the year before.

I continued to explore and compare applications build with the Qt and GTK toolkits and these proved to be my most popular articles. But I also decided to take a little peek into the bigger world of *nix and have a shy first look at Hurd and BSD. My focus completely remained on Linux, though (little did I know that this would come to an end in the future!). Then I dug into package building and learned a lot by trying to update an old and no longer supported Linux distro. Finally I got my domain elderlinux.org and made the first step towards my original goal: Building my own Linux distribution (you have to have done that once, right? And if only for learning purposes).

2014

In 2014 things started to decline. The page hits raised slightly to over 6,800 but that was it. I published 14 posts, but all top ten most popular ones were written in previous years. I didn’t notice that back in the day, though. I managed to get a wide variety of topics covered, including my first post on hardware (writing about the new RISC-V platform that I still keep an eye on).

The most important achievement of the year was that I completed my Arch:E5 project. My own distribution was Arch-derived but did a lot of things different. It used the de-blobbed Linux libre kernel, was based on a different libc, replaced systemd with runit and used LLVM/Clang as the default compiler among other things. It also used a more modular repository architecture compared to mainline Arch Linux. I took this project pretty far: In the end I had a nice self-hosted distro that even came with two desktop environments to choose from. I learned a lot by doing this but since nobody else seemed to be interested in it (I didn’t reach out on the Arch forums or anything, though, to be honest!), I ended the project, continuing to explore other things.

2015

This was the year things changed. Page hits dropped: With about 6,500 hits fewer people visited my blog than even in 2013. I only wrote one post per month (with the exception of April where it was one April fools article and another setting things straight again). Only two posts of this year made it to the top 10 of most popular posts: One about the “Truly Ergonomic Keyboard” (which obviously brought some people to my blog who would probably not be interested in most other articles that I wrote) and another one that was a “FreeBSD tutorial for Linux users” (that received unusual attention thanks to being featured on FreeBSDNews).

I didn’t intend it to, but 2015 was the first year on the blog that was totally dominated by *BSD topics. Since I had started to seriously explore FreeBSD and OpenBSD, this looks like a natural thing. I wrote an April Fools post about Arch Linux’s Pacman coming to OpenBSD and then tried to prove that actually works. Then a friend asked me about FreeBSD and I decided to write a little introduction series. And then the year was more or less over.

2016

After the disappointment of declining public interest in my blog I didn’t expect much from 2016. Especially as I had been venturing deeper int *BSD territory – and liked it enough to continue writing about it. This was obviously even more niche than Linux and how many people would want to read that stuff, especially from a beginner? I was in for a surprise: the blog got more than 7,100 hits that year with four new posts (all of which were featured on FreeBSDNews) making it into to top 10 this time! I had hoped to reach 7,000 hits in 2014 and after it looked like things weren’t going in a good direction, this was a pretty rewarding experience.

I wrote about various *BSD topics: A howto on setting up a dual-boot FreeBSD/OpenBSD with full disk encryption, a little comparison of documentation in Linux and (Free)BSD, a short introduction to Vagrant and a series on getting started with Bacula on FreeBSD. And finally in December an article on using TrueOS for over three months as my daily driver. This post would spark a lot of interest in 2017, making it the top ranked popular post at the time I write this.

2017

In the first half of this year I have already written 14 articles, including two series that a lot of work went into: The adventures of reviving and updating an ancient FreeBSD 4.11 system with Pkgsrc and building a home router with OPNsense/pfsense. And now after only 6.5 month page hits had already climbed up to over 6,700! Recent 3 month have all totalled in more that 1,000, a mark that I had never reached before.

And that’s all before FreeBSD News, Lobsters and even DragonFlyDigest linked to either my pfSense vs. OPNsense article or even to the whole BSD home router series! That made the stats really skyrocket over the previous two weeks. It definitely looks like there are quite some other people out there that don’t think *BSD is boring!

Current stats

Daily blog stats 07/2017

Before the great rush I was receiving about 20 to 60 page hits each day. The new record is now 425 hits on Jul 18 after Lobste.rs picked up the pfSense vs. OPNsense comparison!

Weekly blog stats 07/2017

Weekly hits were between 140 and 370 between Jan and Jul. And then there was this week that saw 1.200 page hits – this is as much as the whole month of May this year and that was the absolute monthly record before!

Monthly blog stats 07/2017

Between January 2016 and June 2017, the blog received 440 (January ’16) and 1.200 (May ’17) hits. And then July happened with over 2.700 hits!

Yearly blog stats 07/2017

The best blogging year so far had been 2016 with 7.100 hits – now at the end of July 2017, this blog has already seen over 8.800 hits. I’m pretty confident to reach the magic mark of 10.000 this time (wow!).

The future?

Of course I cannot say for sure. But I’ve found my place in the FreeBSD community and made a comfortable home with GhostBSD. After becoming part of the small team that develops this OS, I’ve faced quite some challenges and without any doubt there are more to come. But it is a great learning experience and being a (albeit small) part of it feels very rewarding.

And even though time is a very limiting factor I currently don’t feel like taking a break any longer! I will definitely continue to explore more BSD and write about it. Next station: Some preparations for an article on using jails on the newly installed OPNsense router (or anywhere else!). Thanks for reading – and see you soon.

Advertisements

Building a BSD home router (pt. 8): ZFS and jails

Previous parts of this series:

Part 1 (discussing why you want to build your own router and how to assemble the APU2),
Part 2 (some Unix history explanation of what a serial console is),
Part 3 (demonstrating serial access to the APU and covering firmware update),
Part 4 (installing pfSense),
Part 5 (installing OPNsense instead)
Part 6 (Comparison of pfSense and OPNsense)
Part 7 (Advanced installation of OPNsense)

Fixing swap

This is the last part of this series of building a BSD home router. In the previous article we did an advanced setup of OPNsense that works but is currently wasting valuable disk space. We also configured OPNsense for SSH access. Now let’s SSH in and su – to root and continue! Choose shell (menu point 8) so that we can have a look around.

# df -h
Filesystem           Size    Used   Avail Capacity  Mounted on
/dev/ufs/OPNsense    1.9G    909M    916M    50%    /
devfs                1.0K    1.0K      0B   100%    /dev
/dev/ada0s1b         991M    8.0K    912M     0%    /none
devfs                1.0K    1.0K      0B   100%    /var/dhcpd/dev

Uhm… ada0s1b is mounted on /none? Seriously? Let’s get rid of that real quick:

# umount /none

How did that happen? This leads to the question: What does our disklabel on slice 1 look like?

# gpart show ada0s1
=>      0  6290865  ada0s1  BSD  (3.0G)
        0       16          - free -  (8.0K)
       16  4194288       1  freebsd-ufs  (2.0G)
  4194304  2096561       2  freebsd-ufs  (1.0G)

There you have it. The second one is all wrong, it’s not meant to be UFS! We have to correct it to have proper swap space configured:

# gpart delete -i 2 ada0s1
ada0s1b deleted
# gpart add -t freebsd-swap ada0s1
ada0s1b added
# swapon /dev/ada0s1b
# swapinfo 
Device          1K-blocks     Used    Avail Capacity
/dev/ada0s1b      1048280        0  1048280     0%

That’s better. Now we need to adjust fstab to make this change persistent:

# vi /etc/fstab

Change the ada0s1b line like this:

/dev/ada0s1b		none		swap	sw		0	0

Ok, we have some swap now, but we’re wasting most of the disk space of our drive. Let’s address that one next!

Preparing the system for ZFS

In the installer we created a second slice (MBR partition) as a placeholder:

# gpart show ada0
=>      63  31277169  ada0  MBR  (15G)
        63   6290865     1  freebsd  [active]  (3.0G)
   6290928  24986304     2  !57  (12G)

Let’s delete it and create a second FreeBSD slice instead:

# gpart delete -i 2 ada0
ada0s2 deleted
# gpart add -t freebsd ada0
ada0s2 added

Now we need to create a disklabel inside and create a partition for ZFS:

# gpart create -s bsd ada0s2
ada0s2 created
# gpart add -t freebsd-zfs ada0s2
ada0s2a added

OPNsense does not load the ZFS kernel module by default. So let’s do that now and also notify the loader to always insert that ko during startup (we’re using loader.conf.local because OPNsense overwrites loader.conf during startup):

# kldload zfs
# echo zfs_load=\"YES\" >> /boot/loader.conf.local

Then we set the ashift. This tells ZFS to adjust to a 4k blocksize which is better for most of today’s drives use instead of 512 byte ones, even though a lot of them will lie to you and claim to have 512 byte sector size. But even on a drive that really has 512 byte sectors, using 4k is better than using 512 bytes on a 4k sector drive. You will only lose some space if you have a lot of very small files in this case. In the other case however, you will hurt performance badly. If you know your drive and you want to use another blocksize, look up how to do it. Otherwise just set the ashift like this:

# sysctl vfs.zfs.min_auto_ashift=12
vfs.zfs.min_auto_ashift: 9 -> 12

With that we’re good to go and create a pool and some datasets.

Pool creation

I’m calling my pool zdata but feel free to name yours whatever you like better. I also enable compression on the pool level and turn off atime:

zpool create -O compression=lz4 -O atime=off -O mountpoint=none zdata /dev/ada0s2a

Next is creating some basic datasets that won’t be used directly (hence forbidden to mount) but only serve as parents for other datasets:

# zfs create -o canmount=off -o mountpoint=none zdata/var
# zfs create -o canmount=off -o mountpoint=none zdata/usr

Let’s move the old log dir and create some new directories:

# mv /var/log /var/log.old
# mkdir /var/log
# mkdir /usr/ports

On with some more datasets:

# zfs create -o mountpoint=legacy zdata/var/log
# zfs create -o mountpoint=legacy zdata/usr/ports
# zfs create -o mountpoint=legacy zdata/usr/obj

To make the system use those we need to add them to the fstab:

# vi /etc/fstab

Add these lines to the file:

zdata/var/log		/var/log	zfs	rw		0	0
zdata/usr/ports		/usr/ports	zfs	rw		0	0
zdata/usr/obj		/usr/obj	zfs	rw		0	0

Once these additional lines are in place, the datasets can be mounted and the old logs transferred to their new place:

# mount -a
# mv /var/log.old/* /var/log/

The directory /var/log.old is no longer needed, but the system currently has some file descriptors open that prevent deleting it. Just rmdir after the next reboot. Speaking of which: It is now a good time to do updates (and change the firmware to the libressl-based one if you haven’t switched already).

BTW: Don’t try to put everything on ZFS! I made some experiments booting into single user mode and moving over /usr and /var. The results were… not pleasing. After doing some reading I found that while OPNsense works well with ZFS datasets, it’s startup process doesn’t cope with ZFS very well. Place its configuration on ZFS and you’re left with a partially defunct system (that doesn’t know its hostname and won’t start a lot of things that are needed).

Full ZFS support is already on the wish list for OPNsense. It looks like that won’t make it into 17.7, but I’m pretty sure that it will eventually be available, making root-on-ZFS installations possible. Yes, pfSense already has that feature in their betas for the upcoming version 2.4. And they even ditched the DragonFly installer and use the familiar BSDinstall which is really cool (dear OPNsense devs, please also take this step in the future, it would be greatly appreciated!).

Is this a good reason to switch to pfSense? It might, if for you this is the one killer feature and you are willing to let go of OPNsense’s many improvements. But there’s one big blocker: If you make the switch you don’t really need to read on. You won’t be able to create jails easily. Why? Because pfSense heavily customizes FreeBSD. So heavily in fact that you cannot even use the ports tree by default! And that is truly a rather sad state of affairs. Sure, a lot of pfSense users actually use MacOS or even Windows and only want to ever interact with the GUI. BSD means nothing to them at all. But if you’re a FreeBSD user it’s pretty annoying if things simply don’t work (and OPNsense shows that there’s no real need to screw things up as much as pfSense does it).

Ports and jails

The OPNsense team provides packages for OPNsense that you can simply install via pkg. However they currently offer only 368 packages, so chances are that you want something that is not there. The FreeBSD ports tree on the other hand means that over 27,000 programs are easily available for you! So since OPNsense is based on FreeBSD (and tries to remain close to it) this is really an option.

On FreeBSD you’d probably use portsnap to get a snapshot of the current ports tree. This won’t work in our case since OPNsense doesn’t have that tool. The other common way on FreeBSD is to use svnlite and checkout the ports tree from the Subversion repo. Again OPNsense doesn’t provide that tool. And it also doesn’t package the full SVN.

So what can we do to acquire the ports tree? OPNsense does provide a git package and the FreeBSD project offers a git mirror of the SVN repositories. But wait a second! OPNsense works together with the HardenedBSD team and they have their own ports tree (based on the vanilla FreeBSD one with some additions). The whole ports tree is pretty big, but we don’t really want (or need) the whole history. Just what various version control systems call “head”, “tip”, “leaf”, … For git we can achieve this setting the “depth” to 1:

# pkg install git
# git clone --depth=1 https://github.com/HardenedBSD/hardenedbsd-ports.git /usr/ports

FreeBSD ships with OpenSSL in base and a lot of ports expect to link against that. We’re however using LibreSSL and so we have to tell the build system to use that by making an entry in make.conf:

# echo DEFAULT_VERSIONS+=ssl=libressl >> /etc/make.conf

If – for whatever reason – you decided to stick to the OpenSSL firmware, you still need to edit make.conf. This is because OPNsense uses OpenSSL from ports which is usually newer than the version from base (that cannot be upgraded between releases for ABI stability reasons). Use ssl=openssl in that case.

The next step is optional, but I recommend installing a tool for dealing with ports. My example is a pretty light-weight port but maybe you want to build something more demanding. Especially in those cases a ports management tool comes in very handy. I suggest portmaster which is extremely light-weight itself:

# make -C /usr/ports/ports-mgmt/portmaster install clean

Once you have it installed, you can install the jail management tool. Yes, I know that I’ve written about py3-iocage a while ago, but that comes with a lot of dependencies and doesn’t provide enough of an advantage over the purely shell based iocell fork. For that reason I would simply go with that one in this case:

# portmaster sysutils/iocell

Alright! Now you have iocage installed and can start creating jails. What services would you want to jail on a small router box that is always on? Think about it for a moment. There are many great possibilities (I’ll likely write another article soon about what I have in mind right now).

Looking back – and forward

What have we accomplished in this series? I now have a frugal little router on my desk that is quietly doing its work. So far it’s just an additional machine between my network and the modem/router box from my ISP. Taking a break from topics directly related to the actual router, I’ll setup some jails (and NAT) next. But then there is a lot more to look into: How to do proper firewalling? What about traffic shaping? How to configure logging? Also VPN and VoIP come to mind as well as NTP, a DNS cache or even vLANs or intrusion detection.

OPNsense places so many tools within reach of your hands. You only have to grab one of them at a time and learn to use it. That’s what I intend to do. And then, some point in the future, equipped with much more solid networking knowledge, I’ll try to replace that box I got from my ISP with my own modem, too. But excuse me now, I have some reading to do and configurations to break and fix again.