Linux Kernel Panel: What's what with Linux today

Linux Kernel Panel: What's what with Linux today

Summary: Some of Linux's best and brightest kernel developers talk about the state of Linux development today.


Napa Valley, CA: At an exclusive gathering at the Linux Collaboration Summit, some of the crème de la crème of Linux developers talked about what’s going on with the Linux kernel today.

Top Linux kernel developers dish on how they work together to make the most popular operating system in the world.

This year, an elite Linux kernel hacker panel was made up of Red Hat's Dave Chinner; SUSE's Mel Gorman; Facebook's Jen Axboe; Nebula's Matthew Garrett; The Linux Foundation's own Greg Kroah-Hartman; and, as usual, moderator Jon Corbet, co-founder of the top hardcore Linux news site,

The panel opened with Corbet pointing out that today "Almost all the people who work on the kernel are paid to do it. Only 10 percent to 20 percent are volunteers. What do your companies expect to get from your kernel work?"

Garrett replied, "Cynically we can't depend on people to do the things we need to get done. I've written some code that helps our needs and then that helps other people. By showing that you're really happy to help other people, they're happy to help you. I help people, they help me." The others agreed.

Corbett then asked Axboe what Facebook got from paying for Linux kernel development. He replied "As we are shown with Open Compute, Facebook's open-sourcing of data-center technology, "It makes economic sense to develop with open source. Facebook probably saves a billion dollars in development costs alone."

Gorman added that with open source development, you're spreading around the risk of development. "If we were just working on our own stuff, it would be like working in an echo-chamber."

Next, Corbet observed that "Many of us work for companies that don't like each other. While some company agendas appear in Linux development, it's not bad. How do we do that?"

Kroah-Hartman replied that "Competitors rely on each other to survive. Marketing and suits can fight, but they know and we [the developers] know that we're in this together." Axboe explained, "Our work runs across companies."

Besides, Garrett added, "These people are my friends. We like each other." To which, Chinner replied "We're working on a common goal. No single company has the expertise to do the job."

That's not to say that everyone loves everyone else in open-source circles. Far from it! Corbet noted that some user-space developers—that is, independent software vendors (ISV)s who build programs on top of Linux—are sometimes very angry with the Linux kernel developers.

Recently, for example,  the PostgreSQL community was mad at the Linux kernel devlopers for making changes that made their database management system run software slower.

Chinner replied that "A lot of people watch we do, but we don't see what they're doing until we break something of theirs. I treat this problem by telling them to tell me about their workloads and what they do. If I don't understand what you're doing, I can't help you. ISVs need to collabrate. This needs to be a two-way street."

It doesn't have to be that way. Kroah-Hartman said he hears about problems like this "all the time. We are very visible. It's easy to find us. Our e-mail addresses are out there. Tell us your problems and we'll see what we can do."

In the case of PostgreSQL, for example, Gorman commented, "We [the Linux kernel developers] felt that Postgres was fine. 350 messages later I knew people weren't happy. Part of that is our problem. We need to be more open and invite them to talk to us."

Some programmers, both inside and outside Linux kernel development circles, also have problems with new Linux additions and features. In particular, Corbett  mentioned that lots of people still hate cGroups. CGroups is a Linux kernel feature used to limit, account and isolate process CPU, I/O, system memory, and other resources.

Kroah-Hartman replied that "We're way past what Unix could do. We blaze new trails." Gorman acknowledged this, but "Some changes aren't helpful. We shouldn't bolt on new stuff to fix old problems." An example of that, in his opinion, is cGroups.

Chinner added that things change and it's impossible to predict how what seemed like a good idea at the time for one case would end up not being right for other cases years down the road. "With cGroups, it was meant for High-Performance Computing (HPC). Today we have completely different use cases."

Besides, Chinner continued, "We don't get things right the first time. We don't know how tech will be used five, ten years down the track. Sometimes we get it wrong."

For example, Kroah-Hartman said, "There was recently a nasty USB 3 bug in the latest test Linux kernels. We broke a lot of people's machines. We fixed it. That was my mistake." All the developers agreed that more testing, particularly automated testing—not just eyeballs—is needed to catch more bugs.

That said, Garrett pointed out, "Linux is the biggest open source project in the history of the world and it has minimal management. It's a miracle it works as well as it does." Besides, Kroah-Hartman added, "We still do it better [build operating systems] than anyone else. We fail less. Give us credit."

Related Stories:

Topics: Enterprise Software, Linux, Open Source, Software Development

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • crickets

    *chirp chirp.*
    • Seems to be ...

      That way all across ZDNET.
      But at least we are not hearing of open telnet ports, compile from source, no sound card support in Linux .. yaddah, yaddah, yaddha
      • At least you didn't type his moniker

        But as the old proverb says, speak of the devil and he'll appear.
        John L. Ries
        • Hi John!

          How are you today?
          • Hi, Mr. Davidson

            Nice horns:

            Rabid Howler Monkey
          • Doing all right

            How about you?
            John L. Ries
      • I think you spoke too soon

        Too early to say this. Give him a day or two and we'll hear his usual rant.
      • Hmmmmm...

        They like to hear about enthusiasts but they are still ignoring users. While there may be sound card support they still don't have good Wifi printer support. Applications are still difficult to install for a typical user. They still don't understand that IT personnel have invested a lot of time and money in certifications and might not have the money or time to retrain or retest. Not forgetting that most applications are not production quality. Is it worth your time googling for hours so that you can install an application on the most widely supported Linux platforms. When will Linux get its act together. Yes Linux is free but then again its not really free.
    • Trolls

      Thump, thump...
  • dsdasd

    I don't know why I even read this "open source" articles
  • Could someone please come up with... easy way to configure systemd? Init scripts can be messy, but it's easy to add a service to system V Init (or even a BSD-style init). Systemd? Not so much.
    John L. Ries
    • I'm glad I'm not only one.

      Having issues figuring out how to make systemd do what was easy in System V.
      I am on a time crunch with Redhat EL7 on the way with systemd, and I have a number of processes that need to be converted.
    • There is no easy way to configure systemd.

      It happens to be impossible.

      It is an application of network analysis with multiple layers of networks involved.

      The approach was studied extensively in the 70s and 80s with a lot of project management tools developed (PERT/Gant... with the emphasis on the PERT network approach). The approach was also discarded in the mid to late 80s as too cumbersome, and prone to error. It also has a LOT of problems with making changes/additions to the net - it happens to be an NP hard problem.

      Now a simple static network isn't too bad - once you spend a lot of time getting it to work.

      But any additions/subtractions to the net cause other problems to suddenly show up that didn't before - and with systemd these show up as timing errors, hangs, shutdown failures (or more hangs), as well as things just not being done.

      Once you get the static network right, it is faster. But getting there is an art, and there are no really good tools for it (some claims to being good tools - for varying definitions of "good").
    • Systemd is a work in progress ...

      We went through this same thing with pulse audio. Its really irritating sometimes, but it seems like the only way new technologies get introduced in Linux. That said, I am using systemd and not really having any significant problems with it. The journal feature doesn't really work yet, but it is quite simple to just load one of several optional logging packages and be up and running with that in no time. Other than that everything is working to my satisfaction on my Mageia system. Pulse audio used to be a big headache but now it works flawlessly, although I have discovered I needed to add some additional sound utilities to expose all of the options. And configuration utilities are a big part of what is missing with systemd. But over time they will come.
      George Mitchell
  • Linux is unstable

    Much of the problems are because Linux emphasizes adding new functionality and the latest and greatest stuff as quick as possible. The kernel is a constant moving target. This makes it difficult to iron out bugs as huge parts of the code is always replaced with new. This means much code is always new and fresh (as evidenced by the stories above how USB3, Postgres, etc broke). The code is never mature and stable. Linus Torvalds have himself said that "Linux does not have a design, and will never have. Linux evolves like nature, which has evolved humans, so we will constantly try new solutions and rewrite everything all the time. In this way, we will evolve faster and become superior."

    On the other hand, BSD emphasizes stability and design. BSD may not have the hottest stuff immediately, but it is stable and never have as much problems as Linux has, when releasing a new version. This is another Linux problem: it is not uncommon that you need to rollback when you upgrade to a new version of Linux, because of unstable ABIs. The API is stable, but the ABI is unstable which may break the device drivers. Just read some forums, there are always threads like "I have upgraded and now my audio/usb/etc does not work". This means sometimes a device driver needs to be recompiled when a new Linux version is released, in best case. In worst case, you need to tweak the device driver and reprogram it. Sometimes Linux developers have abandoned the device driver, and you need to find another Linux developer who might be interested in modifying the driver. If there are 10.000 device drivers out there, and 2000 Linux developers updated device drivers eight hours a day, they will not be able to catch up with Linus Torvalds when he releases a new kernel version. So all hardware vendors, need to keep different device driver versions for every Linux kernel version, eating up programming resources. I'll leave you with this link: if HP, one of the largest OEMs on the entire planet, can't get Linux to work without running their own fork, what chance does the rest of us have?

    Thus, the driver version is tied to specific Linux version. It is as if Crysis v1.23.5 is tied to Windows XP SP3 and if you upgrade Windows, you need to get another version of Crysis. Windows has stable API and ABI within a version. So if you upgrade from Windows 7 to SP1, you will never have the problems of your audio stops working, which you may have with Linux.

    Thus, another of Linux problems is the Linux device driver model which is broken (because it has no design). No other OS behaves like this, even OS/2 has stable API and ABI. Just look at how many sound APIs there are in Linux, are they four now? Or more? This makes the Linux code messy and bloated. But hey, Microsoft has for ages boasted about how good and stable Windows is (which is not), so Linux kernel devs can do the same for Linux I guess.
    "We're getting bloated and huge. Yes, it's a problem," said Linus Torvalds....Asked what the community is doing to solve this, he balked. "Uh, I'd love to say we have a plan," Torvalds replied to applause and chuckles from the audience. "I mean, sometimes it's a bit sad that we are definitely not the streamlined, small, hyper-efficient kernel that I envisioned 15 years ago...The kernel is huge and bloated, and our icache footprint is scary. I mean, there is no question about that. And whenever we add a new feature, it only gets worse."
    "[Linux] is terrible," De Raadt says. "Everyone is using it, and they don't realize how bad it is. And the Linux people will just stick with it and add to it rather than stepping back and saying, 'This is garbage and we should fix it.'"...IT company CEO: "You know what I found? Right in the [Linux] kernel, in the heart of the operating system, I found a developer's comment that said, 'Does this belong here?' "Lok says. "What kind of confidence does that inspire? Right then I knew it was time to switch."

    Prominent Linux kernel developer Andrew Morton complains about the declining quality of the Linux kernel. His words:
    Q: Is it your opinion that the quality of the kernel is in decline? Most developers seem to be pretty sanguine about the overall quality problem. Assuming there's a difference of opinion here, where do you think it comes from? How can we resolve it?
    A: I used to think it was in decline, and I think that I might think that it still is. I see so many regressions which we never fix.",14495.html#comments
    "The Linux kernel source code has grown by more than 50-percent in size over the past 39 months, and will cross a total of 15 million lines with the upcoming version 3.3 release....In an interview with German newspaper Zeit Online, Torvalds recently stated that Linux has become "too complex" and he was concerned that developers would not be able to find their way through the software anymore. He complained that even subsystems have become very complex and he told the publication that he is "afraid of the day" when there will be an error that "cannot be evaluated anymore."

    Prominent Linux kernel hacker Con Kolivas compares the Linux kernel scheduler code to Solaris:
    "...The summary of my impression [after reading the Solaris code] was that I was... surprised. Now I don't claim to be any kind of expert on code per-se. I most certainly have ideas, but I just hack together my ideas however I can dream up that they work, and I have basically zero traditional teaching, so you should really take whatever I say about someone else's code with a grain of salt. Well, anyway, the [Solaris] code, as I saw it, was neat. Real neat. Extremely neat. In fact, I found it painful to read after a while. It was so neatly laid out that I found myself admiring it. It seems to have been built like an aircraft. It has everything that opens and shuts, has code for just about everything I've ever seen considered on a scheduler, and it's all neatly laid out in clean code and even comments. It also appears to have been coded with an awful lot of effort to ensure it's robust and measurable, with checking and tracing elements at every corner. I started to feel a little embarrassed by what we have as our own kernel. The more I looked at the code, the more it felt like it pretty much did everything the Linux kernel has been trying to do for ages. Not only that, but it's built like an aircraft, whereas ours looks like a garage job with duct tape by comparison.As an aside, I did google a few terms they used which I hadn't seen before, and I was more than a little disappointed to find patents regarding the names... Sigh....Now this would be a great time to take my comments out of context without reading on. The problem is that here was a scheduler that did exactly what I hate about what the Linux kernel scheduler is becoming. It's a monstrosity of epic proportions, and as far as an aircraft goes, it's like taking an Airbus A380 on a short joyride if you're running it on a desktop. It looks like a good, nay, great design for a massive airliner. By looking at it alone, I haven't got the foggiest what it might run like on a desktop. Now since I'm full of opinion and rhetoric and don't ever come through with any substance (maybe writing my own scheduler invalidates that?), I'm going to also say I can't even be bothered trying it, for you to confirm your suspicions about me....the Linux kernel (scheduler) suddenly looks like the Millennium Falcon. Real fast, but held together with duct tape, and ready to explode at any minute...."

    Linux hacker complains of the high code turnover rate, all the code is rewritten frequently, without testing the code. This leads to people breaking their systems when they upgrade, as explained in the article.
    "The [linux source code] tree breaks every day, and it's becomming an extremely non-fun environment to work in. We need to slow down the merging, we need to review things more, we need people to test their f--king changes!"
    • Nice try

      Your articles are all old 2005-2009 and are dwarfed by all the positive articles about Linux available on the internet. Nice try but go back under the bridge from whence you came.
    • Isn't Linux great!

      Your long post is interesting, but mostly out of context.
      You appear to assume these issues are unique to Linux, which they are not. We all know about these things because Linux is the only OS developed completely in the open for everyone to see, and contribute.

      For example, when Linus says the kernel is bloated, he is talking about bloat relative to earlier Linux kernels. Is it bloated compared to other OSes? No.

      Another example: Device drivers: All pre-Windows Vista drivers don't work in Vista, 7 or 8. Why? For the same reason old device drivers may not work in current Linux: Some calls made by the driver have been removed or replaced because the call was either insecure or flaky, or whatever.

      Yes, Linux development is different than any other OS, and I am thankful for that. Does it cause some problems, yes, but it fixes more problems than it breaks.

      One final note: Employees at MS and Apple sign NDAs assuring we never hear anything bad about those OSes from insiders, and none of us will ever be able read programmer comments like "Does this belong here" in OSX, iOS or Windows code, but those comments are in there.

      Maybe you prescribe to the school of thought of "What you don't know, can't hurt you.", but for Linux developers, knowledge is king.
      • Wrong

        "....Another example: Device drivers: All pre-Windows Vista drivers don't work in Vista, 7 or 8. Why? For the same reason old device drivers may not work in current Linux: Some calls made by the driver have been removed or replaced because the call was either insecure or flaky, or whatever...."

        This is wrong. Vista drivers works with Windows Vista. MS has never guaranteed that Vista drivers will work with XP or Win7. MS guarantees that within a Windows version, the drivers will work. If you have Windows XP drivers, they will work with WinXP, WinXP SP1, SP2 and SP3. Period. If you have Win8 drivers, they will work with Win8.1 or Win8.1 update1, Win8.1 SP1 and SP2 and SP3, etc. Windows has stable API and ABI within a version: WinXP or Vista or Win7 or... Just as every OS works.

        OTOH, Linux has unstable ABIs within a version. V2.6.35 drivers might not work with v2.6.39. Linux needs to have stable ABIs within, say, v2. So all drivers will work within version 2.xx. This will make drivers work with v2.4 or v2.6 or v2.1. And within v3.xx all drivers should work - just like very other OS. So Linux should have stable ABIs within v2.xx and within and within etc This would Linux a lot more reliable, and behave more like other OSes.

        Another problem with Linux is that it used to over commit RAM. I dont know if it is fixed now, but if you start a program Linux will allow you to do that, even if RAM is lacking. Say your new software needs another 1GB RAM and you dont have it. You only have another 200MB. Linux will nevertheless start the program and lie to the software "yes, I have 1GB RAM free, go ahead and run". When RAM runs out, what will Linux do to free RAM? It will _randomly kill a process_. Yes, that is true. Imagine Linux kills the important database server in the enterprise server. Or another crucial process. Do you know understand why Linux is regarded as unstable? You start up too many processes, and suddenly some crucial processes has been killed, which makes Linux unstable. No, joe, that is not good. Other OSes will stop you from starting up software, it will say "no, you dont have 1GB RAM free, stop". Which leads to the OS will not have to randomly kill off processes. Read more here.

        There are LOT of other short comings in Linux. Mainly because it has no design. And large parts of the code is always new and fresh, it is never mature and stable. It is said that you need Windows Service Pack 1, before a new Windows version is stable - because in SP1 all bugs have been ironed out and the code is mature. Now imagine that they rewrite huge parts all the time - when will the code mature? Never.

        Which is evidenced by the fact that new Linux versions breaks peoples installations all the time (read about USB3 above, and postgres database, etc). There are numerous stories on the Linux forums how a new upgrade broke something. All Windows fanatics ("yes, Linux is very stable, it only crashed twice last week") has switched to Linux ("Linux is very stable, i only had to roll back my upgrade twice last month because USB3 broke - that is stablilty incarnated"). Have you read about the girl Linux developer that found a bug in the USB2 stack recently? The bug was 5 years old and all the time the Linux devs blaimed the users - "no there are no bugs, you are doing it wrong, it is your hardware that is faulty, you dont know how to use Linux, etc".

        There are MANY shortcomings in Linux. I could post many more links on this. Switch over to BSD instead, if you value stability and design. It might be a bit slower (2-3%?) but it is stable and a true server OS. Linux evolved as a desktop OS, trying to get into servers. Good luck with all those design issues. I know many sysadmins that would never touch Linux with a five feet pole (nor Windows). They loathe x86 servers - because they are buggy. x86 has a legacy from 8-bit cpus with huge bloat and legacy instructions. Read here on how bloated x86 are. If you rebuilt a new cpu from scratch, it would be twice as fast as x86 and use half the energy.

        These sysadmins prefer SPARC, POWER and IBM Mainframes. Those are true servers. Stable shit that never goes down. Not some Windows crap on x86, or unstable undesigned Linux on x86.
    • Depends on your distro

      You can have everything from ultraconservative (like Debian stable) to bleeding edge (like Gentoo). It all depends what trade-offs you're willing to make.
      John L. Ries
      • EXACTLY!!!

        There IS such a thing as LTS (long term support) versions out there that feature BACKPORTED older kernels that are compatible with older drivers and older software in general AND are compatible with a wide range of proprietary Linux software. In the extreme, that is what ENTERPRISE versions of Linux are all about. These versions can be as stable as BSD, but do not provide all the latest bells and whistles that cutting edge versions do. Its all about choices. The problem is that too many users like to have their cake and eat it too. And this is not anything new to the world of software, I saw the exact same issues with Unix years ago before Windows ever came on the scene. Unix always seemed to have things that were broken and never seemed to get fixed. We were always trying to work around something or other. And Windows has always had its own problems, like when you would install and new application and it would overwrite some library and break another application. Its just part of dealing with software.
        George Mitchell