Lennart Poettering on systemd’s Tumultuous Ascendancy – InApps Technology 2022

Lennart Poettering is one of those few extremely prolific and brilliant engineers who have literally changed the design of Linux-based operating systems. In this interview, we discuss his first interaction with the Linux community, why Canonical’s Upstart initialization daemon failed to gain traction and why Poettering created his own initialization software, systemd. He also talks about what he feels is wrong with Docker containers, and much more.

PulseAudio and Avahi

Poettering has been involved with software for a very long time, even before his university days. One of the first projects that he did was Avahi which was a reimplementation of Apple’s Bonjour network service discovery framework. He then wrote his university thesis about that project.

“Actually I wrote it about OLPC (One Laptop Per Child) and service discovery in the mesh networking stuff that OLPC had put together. That was just my thesis; what I proposed there never actually became real because of course why would I do that. I already had to write about it,” said Poettering.

“Why would you use containers if you want full privileges, it makes no sense to me.” — Lennart Poettering

In part, Poettering wrote Avahi to support a Unix sound server he wrote, PulseAudio, first released in 2004.

“One of the reasons I wrote Avahi was I wanted to network audio with PulseAudio and I had the impression that it would be useful if I could actually discover the other sound service on the network automatically. That’s why I was looking into service discovery and found Bonjour, but I didn’t really like the sources. I thought okay I’ll write this on the weekend. It took a lot longer than a weekend. Then I worked on PulseAudio which was a huge amount of work. It was my first contact with pushing thing in the Linux community.”

Working at Red Hat

Poettering’s work on PulseAudio also landed him a job at Red Hat, where he currently works. He recalls that back when Pulse Audio was released, there were very few people in the Linux world working on audio, primarily because the audio was not a market for Linux vendors to monetize from. Poettering’s work was good enough to get Red Hat’s attention and he got hired to work on audio. “Red Hat actually wanted me to work on PulseAudio, which was very comfortable because I was the only guy doing audio at Red Hat at that time and still right now we’ve only got one guy at Red Hat who does that. I had a lot of freedom,” he said.

End of Upstart and Beginning of Systemd

It was PulseAudio and Poettering’s interest in service management that led to the conceptualization of systemd. “Since PulseAudio is a service I kept thinking about service management. I even started talking about it with a colleague at Red Hat.” As prolific as he is, Poettering started writing a project called babykit.

About that time, Scott Remnant from Canonical put together Upstart which, Poettering said seemed to be the future of service management in Linux. While many consider systemd as Red Hat’s answer to Upstart, some accuse Red Hat of sabotaging it. That was never the case, Poettering said.

Red Hat, in fact, adopted Upstart on its enterprise offering. But as Upstart matured, Poettering saw some fundamental problems with the project. Those objections were never political, it was always technical, he said. “Upstart was great, but I think in the end it never actually went anywhere, it moved very slowly,” said Poettering, “Even though the code was very good we figured out eventually that the concepts and logic behind it weren’t really convincing. We couldn’t even deal with a number of the most basic parts of the bootup process. For example file system checking, it couldn’t schedule that properly. It dispatched all that to an external project called Mountall.”

That’s when Poettering decided that Upstart was not going to be what he was planning with service management. The whole concept behind Upstart was broken, he felt, so there was nothing to be salvaged, and a new project was needed. That’s when he created systemd.

Contrary to what people think, Poettering’s managers at Red Hat didn’t like his proposal. “I think it was political partly in there because Upstart was the only thing that Canonical was actively maintaining, it was part of the general stack. Red Hat management at that point liked it. They wanted to keep Canonical involved. With systemd, we ended that because after systemd was adopted by Debian that was the end of the story for Upstart. Canonical lost interest and the users lost interest.”

Systemd Solving Problems for the Enterprise?

So what problem is systemd solve for enterprise customers? “The problem that systemd currently solves is really at the core: service management. Regardless if you have an enterprise system or you have an embedded system or a desktop system, monitoring computers really need to run many parts, many programs at the same time. There are lower level components that we tend to call services that just provide functionality to other programs and then the higher other ones,” he said.

“I have a suspicion that a lot of people are using containers where they probably shouldn’t”–Lennart Poettering.

“With systemd we focus on the lower level ones and managing those, that means bringing them out at boot and making sure they stay running. Making sure that they automatically start if somebody asks for them. Making sure that they automatically start if the specific hardware is plugged in. Making sure that they react to certain system data changes and that the administrator can interface with them. That’s systemd’s core business: service management.”

It’s a complicated task actually. At boot up, a system has to work with about dependencies, to make sure that they schedule the services in the right order. That actually can get quite complex. “In systemd we have this dependency tree and cyclic dependency checks to make sure that what we execute there actually makes sense and that the developers can schedule things at boot in the right order and systemd can actually deal with it,” he said.

Now Poettering is working on bringing service management into the next generation of the operating system.

Systemd and Containers

Poettering is critical of the way people are loosely using the term and technology container management. “A lot of people I would say misuse, other people might say use, container management where they’re actually looking for service management. The lines between container and service management are pretty thin, but I’d always draw the line at isolation,” he said.

Poettering explained that a container tries to create a very isolated environment for programs, whereas service, in the traditional sense, is about integration that runs directly on the host and can see the host runtime objects.

According to him, one of the most important traits of containers is app packaging and delivery. “They are basically what I’d like to call an RPM 2.0 [referring to the RPM package manager]. A way how people can package up stuff and deliver it on systems. Looking at that I have a suspicion that a lot of people are using containers where they probably shouldn’t. They are interested in the image delivery part but not so much in the isolation part. Or where they are actually interested in the sandbox functionality but still not in the complete isolation.”

Poettering is highly critical of the concept called super privileged containers, which are basically containers that run at full system privileges and see the entire operating system. “For me, that’s a big contradiction, why would you use containers if you want full privileges, it makes no sense to me. You’re not talking about containers if you do. If you do, then you care really about the shipping of the images.”

“One of the most well-known concepts that people often cite about Unix is that everything’s a file. I personally think that’s a complete bullshit thing.” — Lennart Poettering

Poettering, as usual, is working on his own solutions and not complaining about the problem. He is working on bringing service management to the next level. He wants to make it easy so users can sandbox system service without necessarily isolating them completely from the host-ware. No need to pretend that a system service is a set of processes completely independent from the host, which is what containers do. System services can still be part of the host and can still maybe see it but not necessarily access things.

The concept behind super privileged containers is really about being able to use the etc kernel directory, local file systems and local devices. “With the sandboxing that we’re currently working on at systemd, you can basically opt in and opt out of these different components individually. Because ultimately the service is still running on the host except that it runs in a slightly more confined environment. Again, not completely isolated from the host and not in its own set of namespaces but still the same namespaces just with more restrictive rights. The difference there between containers and this kind of sandbox services is really about integration and isolation versus sandbox if you understand what I mean.”

Systemd and the Unix Philosophy

Systemd is de facto initialization system on the operating systems of all three enterprise Linux players: Red Hat, SUSE, and Canonical. Leading kernel developer Greg Kroah-Hartman has praised systemd. “Systemd does a lot of things right,” Linux creator Linus Torvalds has stated. But others are quite critical of systemd, criticizing the software’s design for not sticking to the Unix philosophy, in which ‘everything’ is a file.

“Most people who are deeply in love with the Unix scheme: they actually know what Unix really is,” he said. “For us at systemd, Unix is a fantastic inspiration. Systemd is very much a Unix project there’s no doubt about that. Then again it should never be taken as a doctrine as being perfectly designed.”

Nonetheless, he criticized the notion that everything should be a file for the OS: “One of the most well-known concepts that people often cite about Unix is that everything’s a file. I personally think that’s a complete bullshit thing. That’s because everything is not a file. My printer is not a file and if Unix pretends that my printer is a file, that’s complete rubbish because a file is a very different thing.”

He said that most people who are programmers will eventually learn about object-oriented programming, hopefully at university or another school. They will find that the most abstract concept that object-oriented programming has, how it sees the world, is generally the object. On Unix it’s the file. “I’m pretty sure that putting the file as the most basic concept is complete nonsense because the file is way too specific; it’s way too different from a printer, from a sound card. I definitely have big issues ‘everything-is-a-file’ concept and I think it’s  misdesigned.”

“Unix is ultimately a design from [before] the 1970’s. I think today’s computers work very different from back then. People who still think that computers work exactly the way they worked in the 1970s probably don’t have much experience dealing with computers and knowing how much they’ve changed,” he said.

He then pointed at another misconception around Unix, that a large codebase such as systemd must be spread out across multiple directories. “People often complain that systemd has one large repository where a lot of components are living in one source code repository. They think that is not Unix and that’s actually the greatest rubbish of all. Because if you look at BSDs, which unlike Linux are actually true Unixes, they all maintain pretty much the entire Kernel and user space in one single source code repository. Contrary to what people think, in Unix, everything ended up being maintained in one repository, just like systemd.”

Linux is actually the opposite of that because here everybody has its own mini-repositories for their mini projects. Poettering said that the Unix model is nicer in that regard and with systemd, they, in fact, try to go a step back towards Unix.

“If you ask me if systemd is Unix then I would say yeah, in some ways, it’s more Unix than Linux used to be before. In some ways, it’s less Unix and I think asking the question in a religious way is completely wrong. It’s as wrong as being fanatic about any other kind of religion.”

He said that what he truly think is that Unix is the greatest inspiration for systemd, but it is not the only inspiration. “With systemd, we really tried hard to look around at other operating systems and while we focus mostly on Unix-like operating systems, we also spent a lot of time looking at Windows to find some interesting idea there.”

InApps Technology is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker.

Feature image via Pixabay.