The Commander-in-Chief of the Linux Stable Branch – InApps Technology 2022

The following interview is part of a series, called Open Source Leaders, where we profile project leaders in the open source IT community, to learn more about how they developed their software as well as the challenges and benefits that come with running an open source project.

In the sometimes-contentious Linux Kernel developer community, the gentle giant of a man Greg Kroah-Hartman is the friendliest face. When you plug a device into a Linux system and it works out of the box, the credit goes to Kroah-Hartman. He travels around the globe, talking to hardware vendors to get Linux to work on their devices.

But Kroah-Hartman was not a Linux user from the beginning: He evolved over time into one of the most influential kernel developers.

Kroah-Hartman has been a techie from a very early age. He started programming on his parent’s personal computer that they bought for home use.

A few years later, Kroah-Hartman was at his first job working on firmware for printers. He attended an embedded systems conference and saw a booth for Cygnus who was offering support for Stallman’s GNU C Compiler Collection (gcc) at the time. His company had just switched to using gcc for a new project so he started talking to the developers there for a while. He recalls leaving that conversation telling those developers that “there’s no way you all are going to be able to survive if you keep giving away your code!”

At his next job, Kroah-Hartman started using SCO Unix as a database server for Oracle on a large system. It was a beast that constantly had problems. In the end, one of the developers wiped a box clean and installed Linux with Oracle on top it. He noted that the same exact machine suddenly worked so much better than it had been earlier and they had much better throughput and no stability issues anymore.

But they couldn’t use it in the final product as they couldn’t get any support for Linux in a way that would make management happy to rely on it, so they went back to SCO. This was in 1996, before any “enterprise” Linux support was really happening. But that first interaction with Linux made Kroah-Hartman curious about it so he installed Linux on his PC at home in a dual-boot scenario and ended up booting Windows less and less over time.

A few years later, he worked for a barcode company where he was writing firmware for barcode scanners and was dealing with the brand-new technology called USB.

The Weekend Project that Changed Everything

Kroah-Hartman learned to test his firmware with all sorts of different operating systems to find bugs in the firmware, as every company had implemented a different USB stack at the time. “Linux didn’t have any USB support to speak of, only a huge out-of-tree patch set from a developer from Spain working on his master’s thesis,” said Kroah-Hartman. “That stack worked semi-well, but not fantastic. I followed the development of the code with interest, and one day [Linux creator Linus Torvalds] announced that he was writing a whole new USB stack from scratch as the large one just wasn’t in a mergeable state. This sparked a lot of development all at once, and I used my devices to test out the new code. I found a few bugs in the stack and submitted patches to the linux-kernel mailing list and they were accepted.”

He shared an interesting story where his wife was going away from a weekend with their infant daughter and she said to him, “Why don’t you write a driver for Linux for that device you have there, you keep talking about it, now you will have the time to do it.”

With that idea, she literally gave the world Kroah-Hartman that we know of today. Kroah-Hartman spent two days holed up with a copy of the Linux Device Drivers book and wrote a driver for a USB to serial port device that he had, which Linux did not support at all.

“I nervously posted the patch for my driver to the mailing list, and in what seemed like mere minutes, I got a bunch of responses back, saying ‘You forgot to do this,’ and ‘What about multiprocessor systems?’ and ‘This is not going to work, fix this and this line.’ I was amazed that I had instantly gotten such good code review for issues that I had never even considered. I was hooked, and knew that I needed to keep doing this in order to become a better programmer.”

Kroah-Hartman was fascinated by this world where people were offering him instant feedback. That was never the case in the proprietary world where he had been working so far. There was no instant feedback loop. So, this experience was exciting and encouraging for him.

“A few years later, my driver had long been merged and I ended up being the maintainer of that tiny driver subsystem. I realized that more people were using the code I did in the evening and on weekends for free than were using the devices that I got paid to work on (it wasn’t a very successful company). When I realized that I could get a job doing Linux kernel work full time, found one, and then never looked back,” he said.

Currently, Kroah-Hartman is involved primarily with the Linux kernel, but he has submitted patches to some small other projects (systemd, LibreOffice, x.org, git) as well.

One of the greatest contributions of Kroah-Hartman to the Linux world is that he has been a major influence when it comes to hardware vendors adding support for Linux. He travels a lot where he talks to hardware vendors convincing them to support Linux.

“All companies have Linux kernel code for their devices somewhere internally, it just takes a bit of effort to get it out into the public and get it merged. By having the developers of the companies, themselves be the owners and maintainers of it, it grows our community and they feel more involved directly in the advancement of Linux so it works well for their devices and their customers,” said Kroah-Hartman.

He works with developers at these companies to help them learn how to be part of the Linux kernel development community. Sometimes the engagement is not that technical in nature. At times, it could be a management issue.

He recalled an experience where developers of this one company were resisting the idea of having their code merged into the main kernel.org tree. Kroah-Hartman failed to understand why would they want to do that as their code had no problems and merging it would only make their lives easier. It would also save the company a lot of money and resources as that code will be maintained by a larger open source community.

“Finally, in discussing things over lunch with a few people, I learned that the CTO of the company had originally written the first code that became their driver,” he said. “The developers were really worried that any remaining code from the CTO would be refactored away by the kernel community and make the CTO ‘look bad.’”

What’s really interesting is that when Kroah-Hartman talked to the CTO about the code he was very surprised to find that his work had survived this long. “He discussed things with his managers and resolved the issue and everyone was happy when the code was eventually merged into the main kernel.org tree,” said Kroah-Hartman.

Fellowship of the Ring

Now Kroah-Hartman doesn’t have to worry about jobs or writing firmware. In 2012, he joined the Linux Foundation as a fellow, just like Torvalds. The foundation sponsors him to do all of his kernel development work that he has been doing for years.

The contract between the foundation and Kroah-Hartman is kind of funny. “My official contract says something like The Linux Foundation can’t tell me what to do, and I can’t tell the Linux Foundation what to do, so it is very hands-off for them and me,” he said. “That being said, I do work directly with a lot of Linux Foundation people and lots and lots of Linux Foundation member companies to help them with kernel development and community issues that arise over time.”

Kroah-Hartman is one of the most senior kernel developers but just like Torvalds, he doesn’t seem to be writing much code himself these days. Most of his work involves code review of the latest development kernel. Kroah-Hartman also maintains the stable kernel releases. This involves taking the bug fixes that flow into Linus’s development tree and merging them into a stable kernel tree that he releases about once a week. Torvalds does a major Linux release about once every two to three months.

Along with the latest stable release, Kroah-Hartman also manages some older “long term” kernel releases that also get bug fixes, which he tries to release on a weekly basis. These older kernels are usually the ones that embedded systems rely on (Android is one big example) and keeping them up to date with the latest security and bug fixes is important for those communities.

“My responsibility with the Linux kernel community is a multifaceted one,” he said. “I help with the new development of the kernel by being a maintainer of several driver subsystems (USB, serial, staging, driver core, and others),” he said. That involves reading code submissions from others and determining if there are any problems with it being accepted. The rate of acceptance is usually about a third of the changes submitted, so lots of changes take many revisions before being in mergeable shape.

Kroah-Hartman is still a developer at heart. He still has a few small kernel projects that he tries work on every once in a while. “Usually these are just small cleanups to different internal kernel functions where I see they can do things better, but every once in a while, I get to work on helping to write new drivers for new hardware. The ‘Greybus’ architecture was one such recent work, which was sponsored by Google for their modular phone architecture. That device never ended up shipping, but the same code and technology is now in some phones used by other companies, and the code is merged into the main kernel tree so that others can also use it,” he said.

Total World Domination

Today, Linux is omnipresent and it’s a result of the right technology at the right time with the right approach towards open source development model by Torvalds. “When I started working on the kernel, the joke we all made was “Total World Domination,” and no one took us seriously. Now look, we actually did it! But now what, everyone relies on us, so we had better not screw it up,” said Kroah-Hartman.

But can Linux screw things up? “I’ve often said that the only thing that will stop Linux is if we do something to ourselves to mess it up. We have more developers and more companies contributing than any other operating system out there. We support more devices and different platforms than anyone has ever been able to do,” said Kroah-Hartman. “And we did it in a way that had never been done by any software project before as well. So, I worry that we will do something to our development process to hurt this, as no external force can really do anything to stop us directly.”

Every year, the top kernel maintainers get together and talk about the development process — what is working and what needs to change. “That self-reflection and change based on that is what I think has enabled us to survive so long,” he said. “As long as we keep doing that, we will be fine.”

Talking about monetization from Linux or building a career around Linux, Kroah-Hartman said, “I wouldn’t have had the career I have had without open source and free software. I’ve worked for small startups using Linux to sell devices, as well as huge companies like IBM who rely on Linux running well on the hardware they sell their customers. I’ve also worked for Novell and SuSE whose business model was selling support for enterprise users of Linux, as well as pre-installing Linux on new computers being sold by large companies like HP and Samsung.”

“Linux was used in different ways by all of these companies in order for them to make money and survive. It is pretty unique in that there are just so many different business cases for how to do ‘monetize’ open source,” he said.

The Linux Foundation is a sponsor of InApps Technology.

Feature image by Sebastian Oliva, CC BY-SA 3.0.