Working on Linux's nuts and bolts at Linux Plumbers

Once a year, Linux's top developers meet at Linux Plumbers and the Kernel Summit to hash out problems and concerns with Linux development. Here's what happens.
Written by Steven Vaughan-Nichols, Senior Contributing Editor

Linux is built on the Linux Kernel Mailing List (LKML) and numerous other more specialized development mailing lists. But email and Internet Relay Chat (IRC) can only get you so far. Sometimes, to get things done, top Linux programmers really need to talk face-to-face with each other. That's where the Kernel Maintainers Summit and Linux Plumbers comes in.

The Kernel Maintainers Summit, Linux creator Linus Torvalds told me, is an invitation-only gathering of the top Linux kernel developers. But, while you might think it's about planning on the Linux kernel's future, that's not the case. "The maintainer summit is really different because it doesn't even talk about technical issues." Instead, "It's all about the process of creating and maintaining the Linux kernel."

That's very interesting to Torvalds because, as he said, "I spend most of my time on the process rather than technical issues." But, for most people, "it's not that interesting."

For the true nuts and bolts of the individual Linux subsystems you need to turn to Linux Plumbers. In the 90s and early 2000s, if a developer wanted to meet with his or her peers, there were no serious developer-oriented conferences. Conferences were one-offs or were mixed in with other broader conferences such as the Atlanta Linux showcase, Ottawa Linux Symposium, and LinuxWorld. 

The developers knew this was a real problem. So, James Bottomley, IBM Research distinguished engineer and a top Linux kernel developer, explained: "We forced our way onto the Open Source Development Labs (OSDL)  board of directors in 2006, with a list of demands from kernel developers to make the organization more useful to Linux kernel developers. One of our requests was that they assist us with a Linux focused conference in the US."

After the OSDL folded in with the Free Standards Group to form The Linux Foundation, the kernel developers got their wish. The first Linux Plumbers conference was held in Portland, Ore., in September 2008. 

Since then, this conference of a few hundred developers -- the latest in Lisbon, Portugal -- has been all about Linux's "plumbing." This is a conference for people who care about Making SCHED_DEADLINE safe for kernel kthreads; testing kernelCI on a broad variety of hardware; and Working with SANE to make IPP scanning a reality. If your eyes glaze over just reading that list, this is not a conference for you.

Instead, it's a conference for the people who know the dirty details about particular Linux kernel subsystems inside and out. In short, it's for "plumbers."

It's also not a show for people who want to learn how these systems work. While developers who are well on their way into becoming kernel subsystem experts can learn from the discussions, this get-together is more for professional peers to discuss ongoing issues with specific Linux subsystems and their issues. As Torvalds pointed out, "Almost half of the people presenting at Plumbers were also speaking at Plumbers."  

Bottomley added, "Plumbers is developer-centric, It's focused on the leading technical edge." 

I've been to many Plumbers and this has always been true. There's no fluff here. Presentations quickly morph into discussions between experts hashing out what needs to be done with such issues as how to bring testing and continuous integration into the Linux kernel development process. This, by the by, as Shuah Khan, a prominent Linux kernel developer and Linux Foundation Fellow, observed, still needs a lot of work. "But, we are making real progress."

For in the end, as the closing speaker Thomas Gleixner, a leading real-time Linux developer said,  "Plumbers brings people together so they can hash problems out."

Related Stories:

Editorial standards