The handful of people who periodically check my blog for updates have no doubt noticed a dearth of content for well over a year and a half (closer to two years, in fact). There are myriad reasons for this, and I’m not writing today to make excuses, but the reality is that sometimes life gets in the way of other pursuits. I’ve been busy, to say the least, but a few curve balls thrown in my direction certainly took the wind out of my sails and stripped my conscious mind of some of its more creative endeavors.

Just a little under a month ago, my father passed away due to complications from COPD after a long and arduous battle with the disease. This culminated in a number of related ailments that required us to put him under hospice care where he endured for nearly eight months. During this time, we developed an incredible friendship with his hospice nurse, and the efforts she put forward to keep him comfortable were no less than astounding. I would go so far as to say that she is one in a million regarding her expertise and background; she was one of the few people Dad would listen to and certainly one of the only healthcare practitioners he trusted during his end-of-life journey. For my part, she became someone akin to the sister I never had. We shared stories, laughter, tears, and loss. Neither Mum nor I would have it any other way—but make no mistake: The path was not an easy one.

I learned that hospice is one of the most under-utilized healthcare programs available. Partially, this is due to the additional stress and work placed upon primary caregivers (usually family). Mostly, though, I feel this is driven by the unfortunate reality that most people simply don’t know hospice exists. It is unfortunate precisely because there are many people who would rather pass in the comfort of their own homes, surrounded by family and familiarity, rather than end their lives the bowels of a faceless, sterile hospital or care facility.

I apologize to those of you I have forgotten to inform of Dad’s passing. I know there are a many of you who have slipped through the cracks. It isn’t because I don’t care; it’s because I’ve since run out of energy. The people who found out first were the amazingly wonderful church family I’ve gotten to know since 2020; my close friends and associates; and immediate family. My social presence elsewhere on the web has served as something of an outlet during these times where I’ve shared a few of my experiences. Frankly, I haven’t felt it was appropriate to place this on my blog until now. There are reasons, of course, and plenty of excuses. But, now is the time.

I’m hoping to pick back up on posting with a bit more regularity. I have a large number of draft items that may or may not see the light of day. When a particular item gets posted depends on how I feel, of course, and whether or not they’re still applicable to today’s world (some are). It’s all going to require effort and motivation to restart a handful of long-neglected hobbies, such as this, but I think I’m getting closer.

The past two years have been something of a personal tribulation: From a fairly long-term relationship ending in a rather unfortunate and ugly manner in 2020, to Dad’s illness taking a turn for the worse in 2021 increasing his required level of care, to having to put a number of plans on hold, to halting work on a startup with a very talented friend and colleague of mine, to facing the death of a family member, to looking into a new line of work writing in a language that I think will be both refreshing and therapeutic, it’s been a bit of an adventure—to put it mildly.

I have a few news items I’m hoping to pass along sooner or later, and I have a more than enough FOSS projects to announce more publicly (leastwise more so than I already have) that I need to revisit and pick back up (especially documentation!). There’s still much to be done. To say that I’m exhausted at this point, mentally and physically, is probably an understatement. Maybe I need a break, a rest, some respite; I don’t know.

Thank you all for your understanding. I’m hopeful that future updates will bring forth a new ray of light into this world!

It’s a popular statement in a world where many distributions have standardized on systemd. “I hate systemd” comes the quip–a statement designed to evoke emotion rather than contemplation. Just a mere mention of Lennart Poettering provokes near-Pavlovian salivation in some persons afflicted by such a malady, and they often haven’t the foggiest notion why beyond it’s different.

This post isn’t intended to be a deliberate defense of systemd, although it can certainly be construed as such. There are valid reasons to eschew its use, just as there are equally valid reasons to praise it for its novelty–and its belligerence to upend established convention surrounding sysvinit. What I hope to accomplish herein is to reduce the emotionally-charged nature of the anti-systemd response and convince proponents that opposition to such views isn’t antithetical to strongly held convictions of traditionalism in the context of UNIX design and philosophy. Whether they come away from this in continued opposition to systemd (or not) is largely uninteresting to me; I’d much rather someone walk away with a better understanding of their own views (and opinions) rather than convince them otherwise.

It has been my experience that systemd opponents typically fall into three camps: The first, people who have a limited understanding of systemd but have read volumes of opinion pieces explaining how bad it is, accepting opinions as truth; the second, people who feel it violates traditional mores of the UNIX world; and the third (and smallest) group who disagree with its adoption strictly on technical grounds. The latter group is unlikely to glean anything useful from this post as their opinions are likely founded on reason and experience (though I may disagree). The former two find themselves in opposition mostly by force of ignorance. Fortunately, ignorance is curable (albeit powerful), and worst case, we can lift those readers into the objectivism of the third category: Informed but still in opposition.

Readers who are mostly indifferent to systemd because they’re uninterested in learning it or are satisfied with the sysvinit (or sysvinit-alike, e.g. OpenRC) that was installed with their distribution are not considered the target audience. Though they may collect useful arguments, either for or against, I don’t expect them to find much else of interest. They may opt to skip this post entirely.

systemd Violates UNIX Principles

The concept that systemd grossly violates UNIX principles is an argument that usually establishes a few key points (erroneously): 1) systemd is monolithic; 2) systemd violates the principle of “everything should be text;” and 3) systemd is unnecessary complex. On occasion, this may coincide with a fourth point that establishes systemd as an unnecessary replacement for some subsystems or forcibly requires the use of its own internal replacements (such as for syslog or DHCP).

Of these, the third and fourth (surrogate) arguments bear the most weight. As such, I will address them last.

Is systemd monolithic? Yes and no. Yes, the PID 1 replacement that fundamentally exists as “systemd” in its binary form comprises a lot of moving parts, but it’s helpful to understand that systemd as a service runner exposes a significant number of kernel internals: cgroups, capabilities(7), namespaces, read-only file system views, complex service dependency management, and more. systemd is complex but it’s not necessarily monolithic.

Indeed, browsing through the number of binaries in a typical systemd installation will expose a wide assortment of services that perform one specific task. systemd-networkd, for instance, can manage network interfaces. Its companion, systemd-resolved, handles resolver configuration via resolv.conf (and honestly does a much better job of it than dhcpcd hooks or resolvconf).

Does systemd violate the principle that everything should be text? Not really. Whenever this gripe surfaces, it’s usually framed in the context of the systemd journal which does store its output in a binary format. It can also be configured to forward its data to syslog, but I don’t think this argument matters. journalctl comes with tools that can transparently read its binary form just fine thank you very much, along with filtering options that are arguably more powerful than your typical less/grep inspection can muster. In fact, to head off the argument that it doesn’t use “standard tooling,” I might argue that syslog doesn’t either–you have to use other user space tools to open and search through the logs; tools that have become a de facto standard through longevity. Nay, the difference exists mostly in the reality that systemd-journald’s output can’t be read by a tool the system administrator might author independently. Leastwise, not without some work.

There is a strength in what systemd does that isn’t easily replicated via syslog. As an example, it’s possible to configure remote hosts to pack up their binary logs and ship them to another location for centralized logging. Yes, you can do this with one or more syslog distributions, but it’s not easy. Compare this with systemd-journal-remote(8), systemd-journal-gatewayd(8) and journal-remote.conf(5), and you’ll learn it’s a simplified process that does little more than upload binary blobs to a remote HTTP API. Bonus: Because it’s a binary format log, you can selectively extract entries from a remote journald instance. Yes, really.

Aside: I recognize some astute readers will find it cleverly ironic that I’d reference three separate manual pages in the context of the claim that remote logging is easier in systemd, whereas their favorite solution requires reading one or two (and ancillary works; web searches; or more). The facts aren’t quite so malleable: In systemd it’s a matter of enabling the correct services and applying the appropriate configuration changes. There isn’t much need to do anything else.

Returning to our original train of thought: The third dispute, “systemd is too complex,” is a matter of whether all this complexity is useful. Most arguments against systemd’s complexity tend to focus on everything it can do rather than what it does, and I think this disconnect stems from a misunderstanding of what systemd actually is. In particular, it’s apparent that the complaint isn’t really about the complexity of the entire corpus of what systemd can do. It’s a complaint following from the belief that it does all of this by default. This isn’t true; many of the systemd services (systemd-networkd, systemd-resolved, and the more recently notorious systemd-homed) are entirely opt-in. They aren’t necessary and they aren’t typically activated by default (though some distributions make that choice).

I would argue that this complaint ought to focus instead on systemd’s reliance on dbus for its internal messaging apparatus. dbus itself is fairly complex (it’s a message bus…), but it also allows systemd to do a lot of its work under the hood by using an existing message passing system rather than reinventing its own (a surprise to some!). Perhaps it could be argued that repurposing a desktop bus was something of an ambitious choice on Poettering’s behalf, but again, it’s not an illustration of a violation of UNIX principles. If anything, repurposing existing tools should be praised as an example of avoiding systemic problems common among Not-Invented-Here adherents!

At this point, I would presume this section has implicitly answered, by proxy, the question of whether systemd’s replacement of conventional tools is strictly necessary or desirable. If not, then I would posit that more competition is good. As an example, systemd-networkd is far easier to configure and start than DHCPcd or dhclient. systemd-networkd has supported DUIDs out of the box for quite some time, and if you examine the contents of /run/systemd/netif/leases/*, you can copy the DUID between installations to retain static IPv4 assignments leased via DHCP. I’ve done this. (Yes, you have to chop the identifier up a bit, but that’s beyond the scope of this essay.)

systemd Integrates Itself too Deeply

systemd is an init (PID 1) replacement process. Deep integration is its job.

Okay, I get it: systemd as a whole is replacing “too many” established packages. As an example, it contains replacements for one or more of the following: DHCPcd/dhclient, resolv.conf manipulation tools, NTPD, timezone management (more on this in a minute), syslog (we’ve touched on this), hostname management, cron replacement, and probably a dozen other things that I haven’t thought about while writing this post.

I would argue this isn’t strictly a bad thing. Is competition against current DHCP clients particularly egregious? I’d think not–you can still use them if you like. systemd-resolved takes much of the guesswork out of configuring a DHCP client and its helpers to properly update resolv.conf. NTP clients are in the same ballpark–it’s entirely opt-in. syslogging, well, we’ve touched on that. And so on goes the list.

Of course, cron replacement seems to be a particularly touchy point. I don’t really know why, because many of the DIY distributions like Arch and Gentoo don’t actually ship cron by default. You have to install one yourself. Then you have to configure it. Although I’ve never made much use of systemd timers, I will say that shipping software targeting systemd means that if I also ship a systemd timer, I no longer have to worry about whether someone has a cron correctly setup and configured on their system at all. This means that if they deploy a bare container image (say LXD) containing a recent Debian or Ubuntu image, they can install my software and expect it to perform periodic tasks as required without any further intervention. systemd timers also do a bit more than cron. Suggested reading: systemd.timer(5).

And, well, let’s be honest. The crontab syntax is just a little eccentric. Sure, it has its charm, and it’s very compact and easy to read (ahem–once you learn it), but are we defending cron on merit or by way of inertia? Contemplate that question for a while before you continue.

So: Timezones. What exactly does systemd need to manipulate the system timezone for? Well, the short answer is that it doesn’t. The long answer is that the traditional way to configure a timezone for the system was to either allow the installer to do it or to manually place a symlink at the location /etc/localtime pointing to the appropriate zoneinfo file in /usr/share/zoneinfo. systemd itself doesn’t actually manipulate this file, but it does include a tool (timedatectl) that does this for you. Admittedly, this tool does many other things, but it also configures the local timezone for you. Is it worth replacing manual invocation of ln(1)? Probably not, but it’s not exactly doing anything new or particularly troublesome either.

systemd is Pushing Toward a Monoculture

I won’t deny this is a real risk. As of this writing, 8 of the top 10 Linux distributions according to DistroWatch.com use systemd as their sysvinit replacement.

(Ignoring FreeBSD, because I know someone will say “but there’s 11 on that list!”–even though it’s not Linux.)

This is both good and bad. I’ll start with the bad.

Any time there’s a software monoculture, particularly one that’s controlled by a comparatively small number of people, there is a real danger of locking ourselves into a specific mindset. The irony is not lost on me that this is the reason for one of the chief complaints against systemd: Traditionalists who cling to their archaic script-based sysvinit lock themselves into the mindset that sysvinit should only be done with scripts. systemd, by virtue of its market penetration, presents a quandary that we may eventually conclude systemd is the only way to do things. This is bad, but the fallout from this is comparatively minor versus, say, macOS which has no other system besides launchd (from which systemd drew substantial inspiration).

For one, I would imagine that there will always be distributions using alternatives to systemd, even if only as a token gesture to appease the naysayers. Gentoo, while it supports systemd as an alternative init system, uses OpenRC by default. Slackware has options to use a traditional sysvinit or OpenRC. Devuan formed as a consequence of Debian switching to systemd (although I believe users can choose among other inits). Void Linux, in what is probably one of the most novel approaches of a recent distribution, thumbs its nose at convention and elects to use runit instead. While it’s true that most distributions–arguably consuming the plurality of users–have standardized on systemd, I don’t believe there’s any real risk that systemd will become a true monoculture.

As a developer, I think systemd is a good thing for a couple of reasons. First, I can distribute unit files with my software that will initialize it exactly as I intended, and it’ll work across any system running systemd. Provided there’s a fairly recent kernel in place, I can have access to each of the features (see above) that can be used to harden running processes. I don’t even need to write distribution-specific initscripts to ensure the process(es) start up as I’d expect. Maintainers can be happier too, because they then only need to copy out whatever unit files were included with the distribution, patch them as they see fit (if necessary–usually not), and go about their business. Second, if I target a system with systemd, I don’t have to worry about specialized packages like supervisord. Instead, I can be reasonably assured that the process supervisor for the entire system will do exactly what I want. Process fails? It’ll restart. Complex dependency chain when distributing microservices? No big deal; systemd will manage that for me.

The best part? All of that comes for free.

Does this mean I don’t think we can do better? Of course not. I’ve heard it stated before that, paraphrasing, “systemd is a disaster, but whatever comes after systemd will be better than what we have now.”

There may be some truth to such a statement, but I don’t think the statement itself is necessarily a reflective (or is that reflexive?) truth either. systemd can be improved upon, sure, but it exposes a large feature set that once required special tooling (C wrappers, anyone?). More importantly, systemd can be used now. Not next year. Not in 5 years. Not in a decade. Now.

I think the push-back against systemd is potentially dangerous, because it risks frightening off people who might come into the field with new ideas. Seeing how Poettering has been attacked personally (he’s even received death threats–yes, really), they may decide it’s not worth the trouble. That would be criminal.

If any time a new idea surfaced, we immediately bowed to pressure and confessed the naysayers were right before so much as beginning the journey, we’d still be a tribalistic people wandering the plains.

systemd was Written by Lennart Poettering–Just like PulseAudio

You got me. I have nothing else to say.

No, really. I’ll be honest–I’ve seen this argument before, and I’ve seen it it presented as a straight-faced counter to explain why systemd is so awful. This argument is anything but objective and seeks to paint systemd based on either the personality (or personalities) behind the project and on his previous work. I’m not sure this is a particularly good argument, because PulseAudio does actually resolve some long standing issues with Linux audio.

If you don’t know why, then it’s plausible you’ve never dug too deeply into PulseAudio before. But, to humor you: If I have multiple audio cards outputting to multiple devices, I can easily switch between them (think speakers and headsets) by changing the sink output from the mixer. That’s something you can’t easily do from Windows without opening the sound options and fiddling around with default output devices. In Pulse, it’s literally three clicks: Open the mixer, right-click the application, and select the output device.

Let’s be honest: “I hate PulseAudio” is absolutely not a valid argument against systemd. It’s intellectually lazy. It’s strawmanning.

Conclusion

systemd’s criticisms are certainly not without their merits, and I think its worth looking at its deficiencies in the context of what it does right as well as what it does wrong. systemd isn’t perfect–no software is–but I think there’s an argument to be made that sysvinit and sysvinit-compatible init systems are long in the tooth. It’s good to see that there are distributions exploring alternatives (again, Void Linux) and that many others have standardized on an init system that solves long standing issues with process supervision and dependency resolution.

Once upon a time, I used to rely on supervisord to manage multiple processes and to provide some guarantees that if an application failed, it would be restarted. Before that, I relied on DJB’s daemontools (hello qmail!). Each of these solved the deficiencies that existed in traditional sysvinits–and did a darn good job of it. Having said that, I think it’s time that PID 1 finally take control over process life cycle management. Windows had this for a long time. It’s time Linux did too.

As I write this, I recognize the truism that exists in the title of this post. Research, real research, takes time, effort, and energy that may be better expended elsewhere. It requires reading, often for extended periods, and is be exhausting. It requires thoughtful insight and introspection into a problem scope that will quickly expand beyond one’s intellectual horizon if unrestrained. It requires challenging mistaken preconceptions–sometimes going so far as to challenge deeply held convictions–and can lead to surprising or uncomfortable conclusions. At its worst, it comprises a seemingly endless path of drudgery; at its best, it can bring new insights.

But is there anything worse than countless hours of research posted freely without gratitude? There is! Derogatory remarks or misinformation posted as established fact whereby it becomes unmistakable that the author hadn’t a clue.

I recently came across a video posted on social media wherein the author spent considerable time whinging over the preliminary Ubuntu 20.10 release notes staking a fallacious claim on their apparent switch away from iptables to nftables. The narrator claimed this was somehow impossible, because it didn’t ship with nftables installed by default.

I won’t link to the video here; I see no purpose in further disseminating or rewarding objectively misinformed YouTubers. What I find incredibly frustrating about posts like his is that in less than 5 minutes’ research, it’s easy to find out what nftables actually is. Hint: It doesn’t replace all of netfilter; it, in fact, uses netfilter under the hood and simply replaces the packet classification framework previous used by iptables. nftables is an incremental improvement over existing tools. That’s it.

In this case, the reason the YouTuber was unable to find the installed tools is because the nftables package is strictly a userspace interface into the netfilter architecture that has shipped as part of Linux since kernel version 3.13. In other words, nftables literally comes in the box.

It’s curious to me that such an egregious mistake could be made when a single search reveals that, what the author thought was a mistaken addition to the preliminary Ubuntu 20.10 release notes (or outright lie?), is strictly a consequence of his understanding. He didn’t know anything about nftables, presuming instead that it cannot exist independently from the installed tools. Apparently, he didn’t know what the tools do either!

I’m aware videos take time to produce, and editing is almost certainly a lion’s share of the process, but failing any sort of due diligence by conducting research into the subject before turning on the microphone and running the screen capture software immediately gives me pause for thought. What other videos did this person post that are full of misinformation or objectively wrong?

Therein lies the rub. Failure to research even a broad understanding of the topic at hand can present a troubling conundrum for the informed reader- or viewership. Without depth of knowledge, what other dangers lurk beneath such troubled waters?

That’s not to say I don’t post things that are demonstrably false. I do precisely that quite often, either by posting conjecture–or my opinions–or by omission. Sometimes, at least on social media, I may be in a hurry to author a quick reply to someone that contains mistakes (and embarrassing typos, but we’ll pretend those didn’t happen). Often on this blog, I forget about posts dating back into my own antiquity that are no longer applicable, and neglect to place a warning that the information contained therein is dated or no longer valid. Essentially, it’s an indirect and unintentional form of dishonesty through omission. Yet readers are more willing to forgo levying judgment for these categories of mistakes if they understand the context (or age) of the post.

For something posted in the here and now, where there’s clearly no effort to conduct even the most basic form of research? That’s almost unforgivable.

Unfortunately, I expect this pattern to get worse, not better. We’ve seen YouTube videos expand to more than 10 minutes in length, because the algorithm rewards long-format content and because monetization options improve as run time increases. What would have taken less than three minutes to present now regularly exceeds 15! Poorly researched videos have greatly outnumbered well-researched competitors for years, and largely because YouTube rewards quantity over quality. Monetization and viewership depends heavily on vast quantities of material such that channels with more content tend to appear in user recommendations more often than those that work to the contrary.

High volume, poor information gets views.

This post isn’t intended to poke fun at anyone in particular (hence why I’m not linking to the guilty party). We’ve all been there. We’ve all been rushed and released things that don’t meet our minimum bar of quality, kicking it out the door and into the world before so much as a single edit. Thinking about it, this post isn’t so much intended itself to be a complaint as much as it is an effort to raise awareness and present a challenge to content creators: You can do better. Don’t post a video to complain about a topic–aim to educate your viewers. If you don’t like something, explain why and dig into the weeds. Raise the signal-to-noise ratio.

Disclosure: I’m not an Ubuntu user, so this post wasn’t authored as a consequence of defending their release notes. I don’t follow Canonical closely and have no idea what their plans are from one release to the next. I also don’t author YouTube videos as of this writing and am yet-another-content-consumer. I have my concerns, and I think we’re heading in the wrong direction.

We can do better.

(Some details have been glossed over to protect the innocent (?).)