Heartbleed: Open source's worst hour

Heartbleed: Open source's worst hour

Summary: People assumed that open source software is somehow magical, that it's immune to ordinary programming mistakes and security blunders. It's not.


Heartbleed was open source software's biggest failure to date. A simple OpenSSL programming mistake opened a security hole in a program that affected hundreds of millions of websites, and God alone knows how many users, who relied upon it for their fundamental security.


We know what happened. A programming blunder enabled attackers to pull down 64k chunks of "secure" server memory. Of course, a hacker would then have to sift through this captured memory for social security numbers, credit-card numbers, and names, but that's trivial.

We know how it happened. German programmer Dr. Robin Seggelmann added a new "feature" and forgot to validate a variable containing a length. The code reviewer, Dr Stephen Henson, "apparently also didn’t notice the missing validation," said Seggelmann, "so the error made its way from the development branch into the released version." And, then for about two years the defective code would be used, at one time or another, by almost ever Internet user in the world.

Sorry, there was no grand National Secuity Agency (NSA) plan to spy on the world. It was just a trivial mistake with enormous potential consequences. 

So why did this happen? Simple — everyone makes mistakes. Estimates on the number of errors per thousand lines of code (KLOC) ranges from 15 to 50 errors per KLOC to three if the code is rigorously checked and tested. OpenSSL has approximately 300-thousand LOC. Thinks about it.

Still, open source programming methodology is supposed to catch this kind of thing. By bringing many eyeballs to programs — a fundamental open source principle — it's believed more errors will be caught. It didn't work here.

This mistake, while not quite as much a beginner's blunder as Apple's GOTO fiasco, was the kind of simple-minded mistake that any developer might make if tired, and that anyone who knows their way around the language should have spotted.

So why didn't they? Was it because OpenSSL is underfunded and doesn't have enough programmers?

Was it because, as Poul-Henning Kamp, a major FreeBSD and security developer, put it, "OpenSSL … sucks. The code is a mess, the documentation is misleading, and the defaults are deceptive. Plus it's 300,000 lines of code that suffer from just about every software engineering ailment you can imagine."

Was it because proprietary software has more paid eyeballs to look for errors? I have two words for that idea: "Patch Tuesday."

So why did this really go uncaught for so long? Why did Google, Facebook, Yahoo, and even the NSA fail to find such a gaping security hole?

I think I know why and I can sum it up with one phrase: "Magical Thinking." We think that because open source code can be more secure, it is more real secure. Wrong!

Everyone just assumed that OpenSSL must be perfectly safe because, well OpenSSL has a reputation for being safe, therefore it was safe. Developers, website developers, security experts, one and all, it seems no one ever thought to actually use those eyeballs that successful open source relies upon to check the code to see if it really was safe.

We were idiots.

We thought that because OpenSSL was open source that everyone was actually using open source methodology to make sure its code was correct. In reality, no one, after that initial approval years ago, ever bothered to check up to see if the code was both right and secure.

The open source method remains as good as ever when used correctly. When it's not, when we simply assume that all the t's have been crossed and the i's dotted, then we're relying upon faith and not testing and that's doesn't work for any program.

Related Stories:

Topics: Security, Enterprise Software, Open Source

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • LOC vs. KLOC

    OH Stevie... It's not 15-50 errors per LOC, it's 15-50 errors per 1000 LOC. That's exactly THREE orders of magnitude off, buddy. Another acceptable way to represent this is with KLOC (thousand LOC). Both of these points are in the mayerdan page you linked to.

    The same page also says Microsoft's (evil closed-source) code is at 10-20 errors per KLOC. So that's kind of embarrassing for the open source community too, eh?
    Big Sparky
    • The effects of being educated by GUI's

      Just want to add.

      I'ts quite easy to polish "errors per KLOC" figures in the closed source world
      To the point where such not can be taken seriously.

      LOC hEh !
    • Can you imagine...

      ...if it really was 15-50 errors per line? It'd be pretty impressive if the code did anything at all
      • re:

        Not too many of the lines of code I write even have close to 50 characters.
        Sir Name
    • just tried the frontpage text prortions of Wall Street journal as a WST.cpp

      in the compiler

      New world record in errors.

      Dunno why . . . . .
      • LOC KLOC issue

        gave it a thought

        Must havve ben somethinng wronng with the spall chicker onnly
    • That's journalism

      I think that the 15-50 errors per line is technical journalism. Not Software.
      Henry 3 Dogg
  • Where have I heard this before?

    "The open source method remains as good as ever when used correctly."

    Oh, I remember! It's just like what always comes out of every conservative's mouth when, after a few years of conservative economics under a Republican president, the bubble bursts, the economy tanks, and we're all on the verge of a new Great Depression. Yeah, the theory is perfect, we just didn't apply it right.

    Also, I'm sure that in SJVN's world, this is somehow all Microsoft's fault.
    Sir Name
    • Many Eyes

      speaks more so to Socialism rather than of Conservatism.

      One of the unfortunate truths of socialism is that 'shared responsibility' always defaults to the lowest common denominator; thus, while the talk of Great Scrutiny in this case is abundant, the reality is that there is little scrutiny at all, especially being that so few get remunerated (a capitalist convention that you seem to be poo pooing here).
      • re:

        And yet, the FOSS community seems to be mostly made up of Libertarians, who are just right wing Republicans who think drugs and prostitution should be legal.
        Sir Name
        • Libertarianism is about personal freedom and . . .

          Libertarianism is about personal freedom and a government based on constitutional law not the new age case law standards that keep changing with the wind. Libertarians might fit better with a republican philosophy because they both are supposed to be in favor of a smaller government.
      • A very American outlook

        I don't think the rest of us get fixated on the "socializers!!!! teabag partiers!!!" Americans scream at each other.

        Open Source isn't a business model (though there are businesses who have created business models around it.) Open source is a development model, one that aims to pool resources. Open Source projects, at their best, work like a trade association, where several companies that need something share the investment in creating it, and contribute their staff to the pool of people working on it. This is Linux's exact model, for instance.

        Then the individual companies differentiate their take on the pooled work with branding, services, and hardware.

        In the case of OpenSSL, the big problem is the whole operation runs on a shoestring. A lot of organizations rely on it, but few have put anything back into the pot. If the community of businesses that rely on Open source (IBM, Google, Yahoo, etc.) decide to pitch in and allow OpenSSL to step up its process, this likely won't happen again.
        • Open source is still safer

          Open source is the outcomes of thousands of experts (voluntees) who continuously contribute to the Open Source systems.

          Close Source only depends on a company's programers.

          So we can still say Open Source is good to use. The available resources and approaches are the factors determining the quality, neither OPen source nor close source
          • You argue quantity, others quality...

            "Open source is the outcomes of thousands of experts (voluntees) who continuously contribute to the Open Source systems."

            Many of those are self-proclaimed "experts" whose contributions are often well-intentioned, but not necessarily backed by any real expertise. We heard from the programmer responsible for this bug, but we really don't even know if he has any specific experience in crypotgraphy. The library he used is banned from the toolset of proprietary vendors (notably Microsoft) - he didn't even have the foresight to look at what standards other developers live by to use as a guide.

            "Close Source only depends on a company's programers. "

            Who are vetted during hiring and assigned resources and positions according to their experience and career goals. It is likely every single programmer working on crypto algorithms at Microsoft has expertise in that area.

            Vetting programmers is just about as important as vetting programs. The Open Source world has very little capability of doing the former.
          • This betrays a poor understanding of open source

            Many of the larger projects are not only created by vetted contributors (the Linux kernel is infamous for being rather petulantly guarded by Linus Torvalds against unwanted interlopers), but the work is often created by assigned and paid developers, whose companies are effectively a project sponsor.

            If you think for a moment, for instance, that WebKit or Gecko is slapped together by home enthusiast 15 year olds, I've definitely got some news for you!
      • Nevertheless...

        ...we do have this mindset that "correct" theories are infallible and if practice seems to suggest otherwise, it's because the theory wasn't properly applied. Conservatives do it, but so do liberals, socialists, and political advocates of every other conceivable stripe. The real problem is that ideological thinking assumes that all of the correct political principals are already known and all that's left to do is to mechanically apply them. In reality, we don't know that much and it's the height of arrogance to act like we do.
        John L. Ries
    • Where have I heard this before? . . .

      The problem with Socialism is you eventually run out of other people's money.

      Please leave the political pondering for a more appropriate thread . . . if there is one for such a comment.
  • Process... Process.... Process...

    I know I'm in a minority, but I've never thought Open Source would be more secure - quite the opposite. The propriety software houses (MS, IBM, so forth) all make a living from the software they sell, as such it needs to be as high quality as possible - as such there are very rigorous testing procedures in place to ensure bugs are spotted and catalogued. Obviously stuff is still missed based on zero-day flaws, but patch Tuesday is actually an example of the process working with regards to known bugs. Back when Win 95 was released there were some 10,000 known bugs - but they were known, and so in order of priority fixed over the regular patch cycle as to hold off product launch would have required waiting until past the year 2000 based on man hours effort to fix. The flaw with any software development is the oversight process, and continuous improvement in testing and managing bugs - closed shops have this fairly under control, but who manages it all for open source (or is it a case of to many chiefs and not enough indians as the say goes).

    One question with Heartbleed though - out of curiosity - what is the other 1/3rd of the worlds web servers running on that weren't impacted?

    So although I might be a minority in saying so, I'll stick with "you get what you pay for" and avoid the free opensource stuff for the foreseeable future.
    • I'll stick with "you get what you pay for"........

      Unfortunately, often the proprietary software that you pay for also has security issues. It is not an either/or choice between open source and proprietary software to have a more secure system. With either approach, we need to "check the code to see if it really was safe". This should be a wake up call to all. Even with the best of intentions, mistakes will be made. If more resources were applied by all of the major players into developing really secure software rather than creating new tiled interfaces, filing lawsuits over "rectangles with rounded corners, or creating invasive eye ware, maybe we would have a fewer incidents like this.
      • you get what you pay for....

        I am not sure if this is a good analogy to open source, but if everyone in my city has the keys to my house, can I sleep at night thinking my house is safe??