Systemd vs. the Linux Kernel – InApps 2022

The incident began with a patch to the Linux kernel intended to limit the actions of binaries run with another user’s privileges, especially root. Different tactics and timings were brainstormed, as often happens on the Linux Kernel Mailing List. Then, in the middle of the discussion, Linux creator Linus Torvalds made a comment apparently about how systemd complicated the effort to find the best solution, showing that the old animosity between Linux kernel and systemd developers was still very much alive, several years after it had peaked.

“You all presumably know why,” Torvalds wrote, and probably most regular contributors to the kernel did. However, for others, the explanation is not all clear — all the more so because it is not what many suppose or perhaps hope.

As you may remember, systemd, now maintained by Red Hat, became standard in most major Linux distributions over the last few years. It is a replacement for init, the first process that runs when a system is started, and the parent of all other processes. It has also been a seemingly been a source of annoyance for many core kernel developers, for an assortment of technical and personal reasons.

Unlike init or upstart, Ubuntu’s in-house init replacement, systemd is also an all-purpose tool for administering system resources and logging activity. While many supported systemd’s efficiency, others thought it broke the unofficial Unix philosophy of keeping programs simple and limited to a single function. A few even condemned systemd as an attempt by its sponsor Red Hat to seize control of the Linux kernel.

However, since its implementation, the opposition to systemd has largely subsided. A few distributions, including Slackware, have not switched to it, and new distributions such as Devuan and the latest release of antiX were started to provide systems using init.

Moreover, for a while, systemd threatened to succeed Mono cross platform .Net framework as the conspiracy theorists’ main target, with the Boycott Systemd Site borrowing its name from the notorious anti-Mono Boycott Novell site. Yet for most users, the issue is dead, no doubt because the changeover is invisible to the average user.

Against this background, Torvalds’ comment seemed unexpected to many. After outlining his preferred tactics for the patch, Torvalds added that he could see several approaches, and “absolutely none of them involve the random ‘take some values from init.’ And yes, a large part of this may be that I no longer feel like I can trust ‘init’ to do the sane thing.”

The mention of ‘init’ clearly refers to systemd, and, since the effect of the patch is especially important when running as root, how it interacts with systemd is a natural concern. However, beyond these obvious facts, the interpretation has depended very much on prior expectations. Some have wondered if Torvalds’ reference was to a recently discovered out-of-bounds bug in systemd, and was a comment on systemd’s code quality.

Other bloggers thought Torvalds was being discrete because Red Hat is a major contributor to his employer the Linux Foundation. Many saw the comment as evidence that Torvalds was opposed to systemd. Asked directly, Torvalds was succinct: “I don’t think I’ve ever criticized systemd. I’ve been very critical of developers who break stuff and then dismiss the breakage or blame others for it.”

In other words, it is the behavior of systemd developers, not systemd itself, that he was referring to — an interpretation that is in keeping with both Torvalds’ general standards and the past interactions between many Linux kernel and systemd developers.

The Backstory

Although often accused of incivility himself, Torvalds rarely makes personal attacks. For instance, in 2014, when Lennart Poettering, systemd’s co-founder, blamed Torvalds for much of the rudeness in open source development, Torvalds made no response. So far as he responded to other similar accusations, he has mostly said simply that that was the way he was.

Instead, Torvalds saves his sharpest criticism for kernel contributors who ignore his basic rules such as breaking a user application with a patch, offering a poorly thought out solution, or refusing to take responsibility for fixing bugs. In 2012, when kernel contributor Mauro Carvalho Chehab blamed a bug on Pulse Audio, one of Poettering’s earlier projects, Torvalds replied:

Mauro, SHUT THE FUCK UP! It’s a bug alright – in the kernel. How long have you been a maintainer? And you *still* haven’t learnt the first rule of kernel maintenance? If a change results in user programs breaking, it’s a bug in the kernel. We never EVER blame the user programs. How hard can this be to understand?

Although Torvalds admitted to journalist Steven J. Vaughan-Nichols that, “I think some of the design details [in systemd] are insane (I dislike the binary logs, for example), but those are details, not big issues.” However, he was correct to assert that he has not condemned the idea of systemd as a whole.

Torvalds’ reservations about systemd came to a head in 2014, when he refused to accept further contributions from Kay Sievers, systemd’s other co-founder. On the Linux Kernel Mailing List, Torvalds told Sievers that he was “tired of the fact that you don’t fix the problems in the code *you* write, so that the kernel then has to work around the problems you cause [….] This has been going on for *years,* and doesn’t seem to be getting any better [….] I am not willing to merge something where the maintainer is known to not care about bugs and regressions and then forces people in other projects to fix their project.”

That systemd Though…

Much of this criticism appears to be legitimate, however. For instance, rather than using Linux’s existing mount system, Poettering added a new mount command to systemd. Similarly, kernel developer Theodore T’so complained on GooglePlus that, instead of writing to its own journaling system, systemd was writing to the dmesg buffer for the kernel. T’so complained that “the dmesg buffer doesn’t belong to systemd. The interface was originally designed for small amounts of information [….] Dmesg was never designed to be a Syslog replacement, which appears to be how systemd is abusing it.”

Commenting on T’so’s thread, Torvalds elaborated with, “What makes me (and others) upset is that, when a bug is reported, with explanations and a suggestion for how to fix it, Kay just closed the bug-report, claiming it wasn’t a bug [….And] this is not an isolated incident. this is how Kay has treated other bugs in the past. Literally months of stalling, closing bug-reports, and blaming other people and projects for problems that he caused, telling others how they should change their projects because he broke something and obviously it can’t be his fault.”

Clearly, Torvalds sees a persistent pattern in the behavior of systemd developers that makes them unworthy of his trust.

Not every kernel developer reacts the same way to systemd as Torvalds and T’so. For instance, Chebab, as a writer of drivers, regularly contributes shims to help integrate systemd with the kernel. Similarly, Torvalds’ lieutenant, Greg Kroah-Hartman, has attended systemd hackfests, where he has clowned around in silly hats with Poettering, Sievers, and other systemd developers.

But for Torvalds especially, systemd developers have consistently violated the unwritten code that makes interactive development possible, behaving irresponsibly so many times that he is wary of what they might do next. If his attitude is heated, who can blame him? After all, what is at stake is his life’s work — and if someone can’t be passionate about that, what can they defend?

The Linux Foundation and Red Hat are sponsors of InApps.

Feature image via Pixabay.