Welcome to the Great Debate
This week we have two of our top debaters, Ed Bott and Steven J. Vaughan-Nichols, ready to argue the fallout after the Heartbleed security flaw. Be ready for some action.
More Topics
Heartbleed: Is the open source development model broken?
After Heartbleed, must open source development change?
Ed Bott
Yes
or
No
Steven J. Vaughan-Nichols
Best Argument: Yes
Audience Favored: No (65%)
The moderator has delivered a final verdict.
Opening Statements
All open source software is not created equal
Ed Bott: I love Open Source software. I find it hard to imagine a world without Apache, WordPress, and Firefox, to name three easy examples.
But the world is not a better place because of a horrible security flaw in another Open Source package, the widely used OpenSSL library that existed on web servers for more than a year. The Heartbleed flaw exposed just about everyone who has ever used the Internet to potentially catastrophic losses.
So what happened to the “many eyeballs” that were supposed to inspect that code and find the bugs? Alas, there weren’t enough funds to pay for those eyeballs. Even after the flaw was discovered and donations increased, there’s still not enough money to sustain development properly. And that story, not enough backing and not enough skilled developers, is true for many, many projects.
There’s nothing magical about Open Source, nothing evil about closed source. What matters is that the project be run by an organization that has the resources to write, test, ship, deploy, and maintain code properly. The idea that any Open Source project can be widely used even if it doesn’t have proper backing has to end.
It wasn't a failure of open source
Steven J. Vaughan-Nichols: Seriously? That's the question? Come on!
Make no mistake about it. Heartbleed was open-source's worst hour. But, it wasn't a failure of open source per se. It was a failure to actually practice open-source development methods.
Every developer makes programming blunders. One of the big reasons why open source works is to quote Linus's law, as proclaimed by Eric S. Raymond, in his seminal essay "The Cathedral and the Bazzar" is that "Given enough eyeballs, all bugs are shallow."
The problem with OpenSSL that let the Heartbleed blunder in is that no one really looked at the code. Heck, it seems not even the NSA looked at the code!
In short, the recipe is still fine, but only if the cooks actually follow it! Of course there are reasons why people don't this. One of the biggest seems to have been simply paying people to look for bugs. To fix that problem the Core Infrastructure Initiative, made up of top tech companies including Amazon, IBM, Intel, and VMware, will now be sponsoring important, but underfunded, open-source projects.
With cash in hand, and people sticking to the true open-source method, I don't see a repeat of Heartbleed coming up anytime soon.
The Rebuttal
Great Debate Moderator
I'm ready.
And I really hope you are too Steven.
Ed Bott
I am for Yes
I've done my homework
Vote for me.
Steven J. Vaughan-Nichols
I am for No
Great Debate Moderator
Inevitable?
Let's start off simple enough. Could Heartbleed have been prevented? Yes or no, and why?
Posted by Zack Whittaker
It was a rookie mistake
Of course this bug could have been prevented. The author of that section of code admits he made a rookie coding error: “I missed validating a variable containing a length," he said. And the one person who reviewed that code missed it as well.
But that’s what happens when you don’t have a formal security review process. That is a requirement of modern software, especially code intended for use in the infrastructure of the Internet. And that level of process requires an owner and a management structure. That doesn’t imply an economic model, but it does require a level of financial and human commitment that wasn’t there in this example.
Ed Bott
I am for Yes
It should have been prevented
It certainly could have been prevented and it sure should have been prevented.
The real problem was the blind faith that hundreds of thousands of programmers, Web site developers and Web administrators put into OpenSSL. If anyone with a programming clue had simply looked at the code they would have spotted it. They then could have reported it and what turned into an enormous security fiasco would have been a minor, short-lived embarrassment.
Open source only works if people actually look at the code. If you've ever done serious skydiving you know that you must check your parachute before you get on the plane. For years, no one checked OpenSSL and so an obvious error hid in plain sight for years. The really amazing thing about Heartbleed is that it seems to have caused as little damage as it did. By rights, millions of "secure" Web sites should have suffered a failure as fatal as your chute note opening on the way down to the cold, hard ground.Steven J. Vaughan-Nichols
I am for No
Great Debate Moderator
Define open source
Which kinds of software, if any, should be open-source by definition? Explain why.
Posted by Zack Whittaker
Wrong question
There are superb open-source programs. There are superb closed-source programs. There are also mediocre examples of each group. The people who write the code and the people who manage the processes that create the end product matter more than whether the code is available for public inspection.
Ultimately it comes down to a matter of trust. Some people will want the ability to inspect code. Others don’t believe that’s important. There’s no evidence that either group is right or wrong.
Ed Bott
I am for Yes
Open source should be everywhere
Every kind of software can, and should be, open sourced. Indeed, I can't think of any programs that aren't open sourced these days.
Oh sure, Microsoft doesn't open source Windows and Oracle doesn't open source Oracle Database, but we have dozens of Linux desktops such as Linux Mint and Ubuntu and databases including MySQL and MariaDB.
Indeed, once you get away from the desktop, it gets hard to find popular programs that aren't open source. The top Web server? Apache. Supercomputers overwhelming run Linux. The top 25 Web sites in the world, by Alexa's count, all run Linux.
What about office software you say? Say hi to OpenOffice and LibreOffice. Games? Valve CEO Gabe Newell is moving his company's entire Steam-based game portfolio to Linux. Mobile devices? You did know Android rules the smartphone market right? Heck, Android-based tablets are now even more popular than iPads.Steven J. Vaughan-Nichols
I am for No
Great Debate Moderator
Open source or closed source?
Is open-source software inherently more secure, or less secure than their 'closed-source' cousins?
Posted by Zack Whittaker
Again, neither
You can make an interesting theoretical argument on behalf of either approach. Publishing source code makes it easier for good guys and bad guys to discover flaws. Closed source programs require would be attackers, with white or black hats, to test the program’s output.
The larger issue is even when open source software is fixed, it can take a long, long time for those fixes to make it into the installed base. One study found that more than 98 percent of open source libraries with vulnerabilities were not updated in a timely fashion, even after those flaws were found and fixed.
This is a fundamental problem faced by every organization that uses Open Source software and hasn’t implemented good patch management practices. Thousands and thousands of important software packages use open source libraries, but unless there’s someone assigned to monitor those libraries they risk being neglected.
Ed Bott
I am for Yes
Open source - if they do their job
It can be more secure if, and only if, it's actually done properly and people actually examine the blasted code. If they don't then it's no more secure than any other code.
Objective studies, such as the one recently done by Coverity, have found that open source code has fewer errors per thousand lines of code than its proprietary brother. And, it's hard to ignore the Communications-Electronics Security Group (CESG), the group within the UK Government Communications Headquarters (GCHQ) that assesses operating systems and software for security issues, when they said that that while no end-user operating system is as secure as they'd like it to be, Ubuntu 12.04 is the best of the lot.
That said, simply because a program is open source doesn't mean that it's magically more secure. On the other hand, if security really does matter to your company, you, if no one else, can actually dig through the code to look for problems. Just trying getting Microsoft to let you look at Internet Explorer's (IE) code for a bug hunt!
Come to think of it, after the latest IE security hole, which hit versions from IE 6 to the latest IE 11, governments started recommending WIndows' users switch from IE to Chrome or Firefox (both open-source programs notch) that may not be such a bad idea.Steven J. Vaughan-Nichols
I am for No
Great Debate Moderator
The proprietary software question
Particularly with security-related code and cryptographic libraries, such as OpenSSL, should these crucial foundations of the Internet be turned into proprietary software? Explain your answer.
Posted by Zack Whittaker
I’m not sure how that would work
Again, there’s nothing magical about open or closed source. But the Heartbleed episode should raise awareness on the part of the maintainers of important programs of how critical it is to question security instead of just accepting it.
My opponent wants us to believe that large organizations will step up to the plate and make this happen. That ignores the realities of software management in large organizations, where this is most definitely a cost center. Even companies like IBM and Red Hat that use open source software extensively don’t treat these projects as if they owned them. There’s no guarantee they will review the entire code base of a particular project, only the portions that directly affect them.
From a selfish standpoint, a lot of businesses might be smart to take a closer look at commercial software (which is usually closed source) for crucial security infrastructure. With a single patch source and a single point of source control, it’s certainly easier to maintain.
Ed Bott
I am for Yes
Are you serious
You want me to trust a black box where I have no idea what's really happening inside it? II Don't Think So.
I believe that security-related code and cryptographic library should be open-sourced. At the same time, we need to make sure that critical open-source projects, such as OpenSSL, aren't allowed to go for years without enough financing. That's why I think it was well past time that such initiatives as the Core Infrastructure Initiative, is getting ready to sponsor vital underfunded, open-source projects.Steven J. Vaughan-Nichols
I am for No
Great Debate Moderator
Should Apple share?
We saw with Apple's 'got fail;' bug that even the major Silicon Valley titans are not perfect and can miss flaws — even with their vast resources. Would Apple have benefited by making some of its most critical security-related code available for public inspection?
Posted by Zack Whittaker
Probably not
The Heartbleed flaw was out there for more than a year before anyone noticed it. Even large, well-run projects like Ubuntu can suffer from really stupid security flaws, like one last month that allowed anyone to bypass the lockscreen by just holding down Enter. (That was fixed quickly.)
Without getting sidetracked into an Apple-versus-the-world tangent, I’ll repeat what I said earlier: This sort of problem is best addressed with a fundamental security review process that is specifically designed to find the most common errors. Whether it’s open or closed source, that is the proven way to deliver (more) secure code.
Ed Bott
I am for Yes
Yes, they would have
The irony is that Mac OS X is, at its foundation, based on the 4.4BSD-Lite2 open-source UNIX and the Open Software Foundation Mach 3 microkernel. Of course, Steve Jobs, a control freak's control freak as well as a genius, wasn't about to let anyone outside his reality distortion field, touch his baby, so it's all moot.
That said, the OpenSSL error was almost as dumb as the goto fail and it went undetected for ages.Steven J. Vaughan-Nichols
I am for No
Great Debate Moderator
Who's responsible?
Who, if anyone, is ultimately responsible for bugs or flaws in open-source? The creators/founders, the community, the end-users (including business customers) of the software, or a mixture of all three?
Posted by Zack Whittaker
The blame game
I’m not sure there’s anything to be gained by playing this version of the blame game.
Should we blame a community because it doesn’t have the resources to properly test code before shipping it? Should we blame customers who deploy software that appears to be stable, solid, and well tested? Of course not.
I do think there is a case to be made for large businesses opening up their checkbooks and paying more to help maintain critical infrastructure like this. But in many cases they’re already paying hefty amounts to Red Hat or Oracle or IBM or another large company for support of that Open Source software. So maybe those companies that are directly benefiting from Open Source should be the ones writing those checks.
Ed Bott
I am for Yes
Take charge
Open source is often said to be created by the community. While some groups, say Debian Linux developers, interpret that rather narrowly, most recognize that everyone, the first creators, the current generation of developers and maintainers, and the users are all members of the community.
Some open-source projects, such as OpenStack, come right out and acknowledge that the users are an important part of the community. That doesn't mean that every user needs to be an expert programmer. Far from it! But, it does mean that if your company depends on a certain open-source program for its livelihood you darn well better have someone on staff who does know the code.Steven J. Vaughan-Nichols
I am for No
Great Debate Moderator
Sharing responsibility
Major enterprise firms don't contribute to the development, testing, and validation of the open-source software they often depend on. Should they?
Posted by Zack Whittaker
In a perfect world, yes
The trouble is, the appeal of most Open Source software is that it’s free. And those big companies are beholden to their shareholders, who are likely to look at any significant line item and say, “Wait a minute, why are you paying for this stuff? I thought it was free?”
And even when big companies choose to participate, they still don’t have unlimited resources, which means they have to pick and choose which parts they focus on and which parts they (continue to) neglect.
The fact that something like the Core Infrastructure Initiative was “inspired by the Heartbleed OpenSSL crisis” is proof that the current Open Source development model is broken and needs fixing. But depending on how many projects are deemed “core infrastructure,” the costs could soar into the hundreds of millions of dollars. I’m skeptical that we’ll see that kind of commitment.
Ed Bott
I am for Yes
Few are willing to pay the price
Oh brother should they ever!
That's been one of the great failings of open source. Everyone wants its goodies, but few are willing to pay for it. That's especially true of those important, but largely invisible plumbing programs, such as OpenSSL, that keeps the Web economy running.
If your business lives and dies by such a program--and in the case of OpenSSL that's about 2/3rds of all e-commerce companies--for your own sake you should pay something to making sure that program will avoid problems that can knock your business out for the count.
The saddest thing about this is that for a program, such as OpenSSL, even a few dollars a year from every business that used it--say the price of a fast food lunch--could have made the difference.Steven J. Vaughan-Nichols
I am for No
Great Debate Moderator
Public disclosure?
Many security researchers and hackers decide to publicly disclose vulnerabilities in proprietary code, often because those companies ignore reports or prioritize them incorrectly. Should open-source code be treated the same way?
Posted by Zack Whittaker
Why not
There’s a double standard at work here, to be sure. Many commercial software companies devote enormous resources to security work and have a great customer focus, but if they don’t dance in response to a security researcher’s demands their customers suffer.
That said, this outcome is unlikely precisely because there’s often no management to shame in an Open source project.
Ed Bott
I am for Yes
It's happening now
Actually that already happens. If you look at any major open-source program's bug reports you'll find users screaming that X problem hasn't been addressed and it needs to be fixed RIght Now.
If the problem is ignored, what usually happens is people start complaining on the project's programming mailing lists. If that doesn't work, blog reports on the problems start appearing and so on with the news of the bug of the day getting ever wider until the developers are shamed into fixing the problem.
And, if that doesn't work, you can always fix it yourself. If they don't accept your patch, then, if you really feel strongly enough about it, you can fork the code. If you were right and it was a major problem you might find yourself the proud parent of a new project. This is how open source works.
In the particular case of OpenSSL and Heartbleed, that's exactly what the OpenBSD Unix developers have done. They've created LibreSSL. This fixes not only the Heartbleed problem but a lot of other rough spots in the OpenSSL code as well.Steven J. Vaughan-Nichols
I am for No
Great Debate Moderator
Elephant in the room
The NSA said it had no knowledge of Heartbleed before it was first disclosed. Considering our tax dollars go to support the NSA's work (including cracking encryption, but also disclosing major flaws when it finds them), did the NSA drop the ball by not finding the flaw in the widely available code?
Posted by Zack Whittaker
Flawed premise
The NSA is not tasked with disclosing major flaws when it finds them. That’s the job of US-CERT. The NSA’s job is to gather intelligence about foreign governments and potential threats to the United States. It can and should share information intelligently with US-CERT, but that’s not part of the core mission.
Oh, and intelligence services from other countries, who had access to the same source code, apparently also dropped the ball.
I’m certain the NSA works very hard with open and closed source projects to find ways to spy on its designated targets. Doesn’t really have much to do with this question.
Ed Bott
I am for Yes
Shame, shame
Here we are thinking that the NSA is the be-all and end-all of online peeping toms and what do we find? They seemingly never found the Heartbleed hole either! Oh the shame of it all!
I'm reminded of the old saying that you should Never attribute to malice that which is adequately explained by stupidity. In this case, both the OpenSSL programmers, untold number of users, developers, and Web site administrators, and yes, the NSA as well, were all guilty of stupidity.Steven J. Vaughan-Nichols
I am for No
Great Debate Moderator
Lessons learned
What is the most practical way of avoiding Heartbleed-like issues in the future?
Posted by Zack Whittaker
Security flaws will always be with us
As long as code is written by human beings and delivered by organizations that have limited resources, vulnerabilities like this one will continue to keep security experts and IT pros busy for a long, long time.
Fundamentally, it comes down to an inconvenient truth: Open Source software isn’t really free. Someone has to pay the costs of development, testing, and management. Part of that cost will continue to come from developers doing volunteer work, but it’s also up to big for-profit companies that benefit from Open Source to pony up as well.
The real solution, the one that my opponent alluded to in his opening remarks, is twofold. First, increase the funding for crucial projects to pay developers properly instead of expecting them to write code in their spare time without compensation. Second, and more importantly, provide professional management and security review processes.
Make those two fundamental changes, and we have a fighting chance to lessen the risk of further incidents like this one.
Ed Bott
I am for Yes
Do open source right
- Check the code, then check it again, then a year from now have some intern check it again. Do you use the code? Check it yourself. Repeat, repeat, repeat. Some bugs really are subtle and can hide for years or maybe the chip architecture has changed and what was perfectly safe in an old server is now just asking for trouble on a new one.
- Invest money in open-source projects. Yes, many people write open-source code out of passion. Love doesn't pay the bills. You need to make sure that the top developers are paid what they're worth.
- Hire people to do the hard work of security auditing. Let's face it. Programmers have loads of fun making new things. They hate to document, they hate to do security audits and boy do they hate squashing bugs in their own or someone else's code. Those all may be dirty jobs, but darn it, they're jobs that have to be done on both open-source and proprietary programs.
Steven J. Vaughan-Nichols
I am for No
- Check the code, then check it again, then a year from now have some intern check it again. Do you use the code? Check it yourself. Repeat, repeat, repeat. Some bugs really are subtle and can hide for years or maybe the chip architecture has changed and what was perfectly safe in an old server is now just asking for trouble on a new one.
Great Debate Moderator
Thanks for joining us
Once again the debaters did a great job and it's up to you to help pick the winner. On Wednesday Ed and Steven will submit their closing statements and I will post my decision. The comments are for your reading pleasure, please add your opinion. Don't forget to vote.
Posted by Zack Whittaker
Closing Statements
Time for a change
Ed Bott
My opponent’s faith in Open Source is admirable. But you can’t run the global Internet as a faith-based organization.
The problem that caused the Heartbleed security nightmare is directly attributable to missing management infrastructure, which costs money. You can’t just ask a couple developers and an intern to audit code. You need a formal security review process and an honest-to-God QA department.
In his rebuttal answers, my opponent admits over and over again that the current Open Source development model is broken. That’s why the Core Infrastructure Initiative is being rushed into existence. But will it get enough funding to do an effective job? Unlikely.
He says, “If your company depends on a certain open-source program for its livelihood you darn well better have someone on staff who does know the code.” Seriously? Tens of thousands of companies use OpenSSL, which consists of “hundreds of thousands of lines of very complex code.” Customers can’t be expected to review every line of code they use.
Unfortunately, Heartbleed probably wasn’t an isolated example. As long as Open Source projects are run on inadequate resources with insufficient management, they’ll happen again. It’s past time for a change.
Related Coverage:
- Coverity finds open source software quality better than proprietary code
- Mistaken Heartbleed clean-up efforts accidentally leaving thousands of servers vulnerable
- In Heartbleed's wake, let's not forget many open-source apps remain vulnerable to attacks
- Internet slowed by Heartbleed identity crisis
- Many sites reusing Heartbleed-compromised private keys
- Heartbleed: Open source's worst hour
Open Source for the win
Steven J. Vaughan-Nichols
Ed and I can argue about the fine details, but at day's end we both agree that if developers use open-source programming methodology correctly you can produce great software. I also believe that proprietary software methods can produce fine programs as well. I just believe that the odds are better you'll end up with high-quality, secure programs if you follow an open-source roadmap for your project.
Successful pure open-source companies such as Red Hat, SugarCRM, and Alfresco, an enterprise content management provider, use all the same software development managements that proprietary software vendors use. Other companies that you may not think of as open source development houses such as Dell with OpenStack and Facebook with Open Compute use both open source and "corporate" development methods.
In short, when done right, open-source combines both open source's virtues with so-called "proprietary" methodologies. It's only in those corner cases, like OpenSSL with Heartbleed, where a program is both popular and under-funded that there exists the real possibility of a major security problem. Now that Heartbleed has made people aware of this and efforts such as the Core Infrastructure Initiative will help prevent major security holes.
So, as far as I'm concerned, open-source is still the best way to develop secure, safe software.
A close race
Zack Whittaker
I must admit, this one was tricky, but I'm going with Mr. Bott. While Mr. Vaughan-Nichols made some good points, I found many of his responses avoided the particular issue in hand — the practical problems with open-source's development approach.
Mr. Vaughan-Nichols nevertheless made some deep-rooted arguments about the open-source development model, notably comparing and contrasting the so-far "one-off" Heartbleed flaw with one of the buggiest closed-source applications on the market, Internet Explorer. And, yes, his argument that large companies that use open-source software should invest in the community they rely on so much almost won me over.
But what clinched it for me was the core of the subject. Mr. Bott's killer closer put the icing on the debate cake. Yes, flaws exist in major proprietary code and closed-source apps. But while sometimes thousands of contributors may help build a better functioning product, the security implications in a world with insufficient management and resources probably won't prevent another Heartbleed from existing.
It was a close race, make no mistake — and the audience agreed by a large majority with Mr. Vaughan-Nichols. But I think Mr. Bott has to take the win in this case.
Posted by Zack Whittaker