Did open source matter for Heartbleed?

Did open source matter for Heartbleed?

Summary: Open source does not provide a meaningful inherent security benefit for OpenSSL and it may actually discourage some important testing techniques. Also, panhandling is not a good business model for important software like OpenSSL.

TOPICS: Security

The ugly episode of Heartbleed has put OpenSSL under more scrutiny than any open source software project ever. At a certain level of scrutiny perhaps any program will look bad, but OpenSSL's on the hot seat because it's OpenSSL that failed in its mission. It's hard to construe these matters in a way that makes OpenSSL or the open source nature of it look good.

But who is this "OpenSSL"? When something goes wrong with a product people want to know who is responsible. Many will be shocked to learn that it's all run by a small group of developers, most volunteers and all but one part-time. Huge parts of the Internet, multi-zillion dollar businesses, implicitly trust the work these people do. Why?

Let's stipulate that OpenSSL has a good reputation, perhaps even that it deserves that reputation (although this is not the first highly-critical vulnerability in OpenSSL). I would argue that the reputation is based largely on wishful thinking and open source mythology.

Before the word "mythology" gets me into too much trouble, I ought to say, as Nixon might have put it, "we're all open source activists now." For some purposes, open source is a good thing, or a necessary thing, or both. I agree, at least in part, with those who say that cryptography code needs to be open source, because it requires a high level of trust.

Ultimately, the logic of that last statement presumes that there are people analyzing the open source code of OpenSSL in order to confirm that it is deserving of trust. This is the "many eyeballs" effect described in The Cathedral and the Bazaar, by Eric Raymond, one of the early gospels in the theology of open source. The idea is that if enough people have access to source code then someone will notice the bugs.

This is, in fact, what has happened with Heartbleed... sort of. Heartbleed was discovered by Neel Mehta, a security researcher at Google. If you look at the vulnerability disclosures coming out of other companies, Apple and Microsoft for example, you can see that Google spends a lot of time scrutinizing other people's programs. They're like no other group in this regard.

But it took Google two years to find it. In the meantime, Google finds lots of security problems in Apple and Microsoft products for which they have no source code. This is because in the time since the formation of the "many eyeballs" hypothesis, there have been huge improvements in testing and debugging tools. Some computer time with a marginal cost of $0 is worth thousands of very expensive eyeballs.

I'd go so far as to suspect that the availability of source makes developers and users discount the necessity of testing that is common on commercial software. I wouldn't be surprised if a static source code analyzer would have found the Heartbleed bug, flagging it for possible buffer over/underrun issues. Heartbleed might also have been found by a good round of fuzzing.

As I said recently, some programs are so critical to society at large that someone needs to step in and make sure they are properly secured. Obviously the problem is money. So why, when this program is so critical, is it being run like it's public TV? Yes, like Blanche DuBois, OpenSSL has always depended on the kindness of strangers.

Are OpenSSL users living in a dream world, like Blanche? There's some truth to this. Is OpenSSL going to crack under the pressure, like Blanche? No, it's not that bad.

The dream world part of it comes from the notion that software is more secure for being open source. Even if this was ever true, and I'm not sure it was, it's surely not anymore.

How about closed source? Is it more or less secure because source code is not generally available? As a general rule, I'd say the answer is no. The key, just as with open source, is how much time is spent by qualified people auditing and testing it. Windows and other major Microsoft products get a ton of attention from outside testers, both black hat and white hat.

Is that same research being performed on major open source projects? Not to the same degree, at least as far as I can see. A large percentage of the vulnerabilities fixed by Apple and Microsoft are reported to them by research companies with bug bounties. I scanned the list of vulnerability reports at HP TippingPoint's Zero Day Initiative, VeriSign iDefense, security-assessment.com and KeenTeam and I don't see any reports about open source projects. Lots of big name software from Microsoft, Adobe, Oracle, Apple, and the like, but no libpng, no Apache, no MySQL, no PHP.  

When vulnerabilities are reported in such programs, it's typically from independents. You can see this by studying Apple's disclosures. You'll see the same in the security updates from Apache although, from what I can tell, most large open source projects don't make it easy to find a list of their security updates and the vulnerabilities fixed in them.

"Cathedrals" like Microsoft and Apple have another advantage in situations like Heartbleed: patch delivery and installation. All Windows users can get their updates from Microsoft and Apple and even consumers have simple, yet sophisticated systems for installing them. There are undoubtedly many OpenSSL installations in forgotten computers, appliances, perhaps even embedded devices, and a fat chance that they'll be updated ever.

So I think it's fair to say, to a point, that OpenSSL benefited little, with respect to Heartbleed, from being open source. It's also fair, to a point, to say that it suffered from being open source if that made it easier not to test the software thoroughly as it should have been. The solution isn't to close the source code, it's to recognize the limitations of open source and test programs like OpenSSL as if no source were available.

For the moment there's little we can do about this. It's virtually impossible to use the Internet without relying on the work of numerous open source projects, the principals of which are unknown to all but a few people and the development standards for which must vary wildly.

Perhaps the conclusion we should all draw is that it works; we wouldn't be using it if it didn't. That's a pretty low bar of accomplishment, and in fact all it means is that it looks like it works. I don't like the idea of entrusting my business to software for which nobody can be held responsible, even if the source code is freely available, but for now I have little choice, and neither do you.

Topic: Security

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • Nice write up.

    I have long said that open source is no panacea when it comes to quality and it all depends on the developers and designers to make a great piece of code. Open source/proprietary? Quality is available both ways. GIGO.
    • Cheap stuff has low quality

      Truth hurts but it's still truth nonetheless. When you don't pay people to produce stuff the incentive to do a good job is not there. So there goes the product quality.
      • Amazingly Balanced

        The article is amazingly balanced on the issue of open vs. closed source. I really like the reluctant conclusion regarding mission critical software.
        • But that's why you go with a a Red Hat or similar

          I don't like the idea of entrusting my business to software for which nobody can be held responsible"

          In the end, if you pay Red Hat for the support (which isn't cheap) for their software, they become responsible for their software.

          They can't hide behind the "well, it's open source and you get what you paid for".
          • Good Luck With That!


            Good luck holding M$, Redhat, Oracle, Symentec, VMWare, SAP or any other software company you can think of accountable for anything. You may want to read one of those License agreements you dismiss as quickly as possible when you're installing software.

            Open Source Software has proven it's it's worth time and time again; is there going to be weaknesses that are uncovered, sure, same with any software. What the Open Source community doesn't do is spend bunch of time looking for someone to blame, they just get it fixed.

          • LOL!

            "What the Open Source community doesn't do is spend bunch of time looking for someone to blame, they just get it fixed."

            Good luck with that one....
          • Cont:

            In the past Open Source has been WAY quicker fixing problems than closed source counterparts; lot less layers of bureaucracy to deal with.

          • Reason why they panhandle...

            See, coding as critical as this is "too big to fail", but the developers (knowing that OpenSSL could somehow take a fall) don't want to be held responsible for that. I think it's a noble cause that they handle all the grueling mind-bending work of algorithms just so that we can order pizza drones from our iPads!
      • what a stupid statement, we love to code and money means s**t to us!

        errr, yeah, like a massive industry's never rush software!! Vista?? The only incentive for them is money!! Compare that to an open source OS such as Ubuntu, the programming community do it out of the love to code... People generally do a better job when doing something they love. You obviously know nothing about software development otherwise you would know how wrong that statement is.

        Are you even aware of the list of know vulnerabilities on Windows 7? Written under the incentive of money... product quality or what ey!?

        • So, let me count...

          Eleven vulnerabilities? I thought you were going to refer us to like hundreds of vulnerabilities. And one of them only applies to IE8 on Windows 7.
          Ehsan Irani
      • True in some cases...

        But in others, even when they get paid nicely sometimes they can still mess things up. There bugs all the time all over in coding, this one just happened to be way up there.
    • Is quality free? Or not?

      I've read the book, "Quality is free", by Phillip B. Crosby. Along with a number of other books on quality (with an emphasis on data quality).

      Even if quality is free in the long run, there are up-front costs for initiating a quality program as well as costs for maintaining a quality program. Management also has to be "all in".

      Some proprietary software vendors (e.g., Adobe Software) have merely used operating system capabilities to sandbox their poor-quality code (e.g., Flash Player, Adobe Reader, Adobe Acrobat). This is damage control and, IMO, is not software quality assurance. Implementing a security development life-cycle project (as Microsoft and other organizations have done) is the way to go as long as it involves more than merely dotting i's and crossing t's.

      Recall that the open source OpenBSD project has managed improve its own software quality since 1996 with significant effect. Although, the very recent OpenSSL Heartbleed vulnerability (the OpenSSL project is not formally part of the OpenBSD project) still managed to give the project a black eye. Perhaps, a bit more funding provided to the OpenBSD project could convince Theo de Raadt to take the OpenBSD project under his wing. OpenSSH is already included in the OpenBSD project.

      Also recall that one of the two GnuTLS primary devs is a Red Hat employee and that Red Hat paid for the dev to perform the recent audit of GnuTLS source code. It would be appropriate for organizations that use, create or support open source software to hire additional staff to help with software quality assurance of various open source projects, especially high-impact open source projects such as OpenSSL and GnuTLS.

      As a user of open source software, the U.S. government also has a stake in open source software quality. Thus, wouldn't it be appropriate for the federal government (e.g., U.S.-Cert) to hire some FTEs to contribute to open source software quality for various high-impact open source projects?

      High-impact open source projects first need to be identified. Once identified, various organizations can decide which open source projects they will contribute to.

      Finally, are there enough software quality assurance professionals available to meet the increased demand from high-impact open source projects? If not, they will need to be trained at uni and on-the-job.
      Rabid Howler Monkey
      • Ugh! "convince Theo de Raadt to take the OpenSSL project under his wing"

        Rabid Howler Monkey
      • Microsoft and security

        "Implementing a security development life-cycle project as Microsoft ... have done"??? And it's worked so well. I installed Windows 7 from scratch on a computer today, and not-so-promptly received 150+ updates, most of which were related to security. Microsoft may have a security development life-cycle, but the best thing Microsoft has done for security recently is to terminate XP support, sending everyone stampeding to a newer and more secure operating system. After all, it is Microsoft that created the mammoth security hole in XP by integrating IE more tightly with the kernel. Once you've done that, you can't improve the OS security unless you undo the tight IE/OS integration. Well, Microsoft did that with subsequent Windows releases, didn't they? But after the heat was off from the US Justice Dept.

        Please, security and Microsoft said in the same sentence is an oxymoron. How many buffer overruns were patched in XP, given the apparent lack of emphasis on security from the get-go? You know what? We'll never know, and there is no excuse for buffer overruns, ever.
        • ben_myers@...: "Microsoft and security"

          "I installed Windows 7 from scratch on a computer today, and not-so-promptly received 150+ updates, most of which were related to security."

          Windows 7 was released in 2009. Install Ubuntu 12.04, released in April, 2012, and let us know how many updates needed to be installed.

          "How many buffer overruns were patched in XP, given the apparent lack of emphasis on security from the get-go?"

          Windows XP was released in 2001. Guess what Linux kernel version hailed from 2001 ... version Many of the security features we take for granted today with Linux were introduced at various points in the version 2.6 kernel, including ASLR which was introduced with the 2.6.12 kernel in 2005. Note that ASLR was introduced with the release of Windows Vista in 2006. Guess what ASLR is meant to protect against ...

          The biggest failure with Windows XP was having the default account as the Administrator. In spite of Windows NT having provided multi-user support from the very beginning. This was fixed, to a large degree, with Windows Vista. Another big failure was not having the Windows Firewall enabled by default with the initial release of Windows XP and with SP1. Unlike Linux, Windows ships with some listening services (esp. RPC) enabled. The Windows Firewall was enabled by default with SP2 in 2004.

          Security features in Windows 8 are a far above those in Windows XP. Microsoft has come a long way since Windows XP.
          Rabid Howler Monkey
    • Foolish statement about Open Source.

      Open Source is the way that a lot of professionals and want to be professionals who have gotten their creds and repaid the debt they owe to the community that helped them get to the top of their field. The proprietary efforts to discredit the progress that Open Source has made, is reflected in the amount of software that the proprietary market has stolen from Open Source. This is all FUD that they are spueing here, Fear, Uncertainty and Distrust. Don't get sucked in.
    • Pretty good write up

      I agree it's a nice write up, but the economic weaknesses of the open source model could have been elaborated upon. It's expensive to do continuous security analysis of code, and you need some revenue model to fund it.

      If the standard SSL/TLS codebase were owned by a commercial entity that charged licensing fees for commercial use, then there would be a steady stream of revenue that could be used to pay for security analysis etc. The code could still be open, but using it commercially would require paying fees.

      Without a revenue model, and with liberal licensing, the only way to fund this sort of thing is via a corporate sponsor. That will only happen if some firm sees a strategic advantage in sponsoring OpenSSL -- i.e. that the net gain to that firm would be greater than the gains to competitors. An alternative would be public sponsorship, if governments view it as important, but at the end of the day, the money has to come from somewhere.
  • I don't really agree with this

    even though a lot of the actual points are correct.

    You note that OpenSSL is probably embedded in routers, devices, and GPSes, etc. and you are probably quite right about that. But OpenSSL's only role in these products is that it is a codebase drawn upon to help produce the product. In theory, the provider of the product is just as on the hook as anyone else to update their product. OpenSSL's role in the product is no different than OpenBSD's role as a place that Microsoft and Apple borrow code quite extensively from.

    I've never bought any of Eric Raymond's "Robert and Elizabeth Browning" love songs to open source. Open Source is just one of many models for producing code - it is neither better nor worse.

    What matters - what has always mattered - is the product itself: the code, and the programs that result. There are exceptionally good open source code bases, heavily scrutinized for vulnerabilities: Firefox/Gecko, and Chrome/Safari/Webkit/Blink spring to mind. Linux.

    And there are some exceptionally good closes source code bases - decades later Photoshop still kicks the GIMP's butt.

    Use what works for you. There are open source based products, such as Oracle's MySQL, Xamarin's developer products, and RedHat that have tremendously good support arrangements.

    People should neither fear nor excessively adore open source. Its just one of many ways of producing code, and as with all methods, mileage may vary.
    • I think you missed the point

      It's pretty clear the author isn't speaking against the Open Source model itself, but rather the somewhat naive view that the model itself, simply by being open, is inherently less susceptible to flaws. This incident has been a pretty clear example of that not being the case.

      As you noted, there's some very good examples on both sides of the open/closed source fence. But proponents of the Open Source are vastly more vocal about the supposed advantages of their preferred model, advantages that sometimes are, and sometimes are not, actually realized.

      Your last sentence seems to be in complete agreement with what I took to be the author's point, so I'm a little confused that you're disagreeing with him.
      • I don't really think so.

        The author closes on this note, " I don't like the idea of entrusting my business to software for which nobody can be held responsible, even if the source code is freely available, but for now I have little choice, and neither do you."

        I think I've made a legitimate case against that. I did open with the comment, "even though a lot of the actual points are correct", so it isn't as though I was ever in utter opposition to the author's views.