Insecurity blues: What I learned from my buggy code

Commentary--It hasn't been a good month for my code. Samba, the project I'm responsible for, has had to announce several security flaws. Unfortunately some of them were in code I wrote. I always do a large amount of soul-searching whenever that happens. There's nothing worse than finding out something you were responsible for is the cause of many thousands of people having to waste their time rolling out patches. It always makes me wonder if the time has come to give up this programming lark and end my days peacefully in management, messing up other people's code instead of creating my own.

It's very educational however to look at the causes of the flaws, and try and learn what we can from the bugs and also our reactions to them. Samba is an old program. The initial code was originally written 15 years ago. At that time, modern security problems such as integer wrap (where adding two numbers together can end up with a number smaller than both of them, due to the fixed sizes of integers that processors deal with), or heap overflow vulnerabilities (where overwriting unallocated memory on the program heap can allow a clever attacker to seize control of a program) were unheard of. We knew about buffer overruns (where reading more data into a buffer than was originally allocated for it can cause a security breach) and denial of service attacks, but 1992 was a simpler, less hostile time for network software development. Most initial deployments of Samba were on networks isolated from the main Internet, by technically advanced administrators who knew how to download the source code from the Internet.

The kind of security bugs we get now are much more sophisticated than those that were originally present in the first releases of Samba. We also treat them much more seriously than we used to. In the early days, we (or someone else examining the code) would discover a security problem, and almost immediately we made an announcement of a fixed version. Those were also the days where binary releases of Open Source/Free Software were unknown, as everyone knew how to compile from source code. At least everyone who we were interested in working with did. Our response time for security bugs was usually measured in hours (depending on the timezone of the person reporting the bug). I still remember the first attempt by someone to sabotage Samba with a zero-day exploit posted directly to our mailing list without any warning. Andrew Tridgell (the original author of Samba) was so incensed that he immediately raced to ensure the very next message on the list was the patch for the bug. He succeeded. These days we rarely get zero-day exploits posted openly to lists. The bad guys who discover exploits keep them secret for their own illicit use, and the good guys go through the currently agreed-upon best practice of sending notice to the Samba developers directly (at the mailing address security@samba.org) and work with us in coordinating the announcement of the problem and its fix.

Our response time appears to be much slower these days, even though internally we still usually create an initial fix for the problem within a few hours of seeing the report of the bug. Having Samba Team members in almost all timezones helps with this speed of response. What slows us down these days is our own success. In order to responsibly release a security fix version of Samba we must coordinate with a large number of vendors who are reselling and shipping our code. Most of them have their own patch release schedule and internal security teams who validate the suggested patch and agree on a time to announce the vulnerability to everyone.

Who discovers our security bugs? These days, it's about 50-50 between internal audits by the Samba Team and other Open Source security people and external security researchers, who make their money winning contracts by proving how smart they are by discovering vulnerabilities in Open Source/Free Software projects. The way we respond has changed also. What we do now on seeing a security bug is immediately audit the entire code-base to discover if there are any similar problems, or even similar coding practices that might cause future problems, and re-write or remove all such code. It takes longer, but is much safer in the long run. If you examine parts of the Samba code you'll find common functions that are known to be insecure simply won't compile if added to our code. A set of automated macros warns of any use of known bad functions. Still, we can't depend on automation for protection. Our current security vulnerabilities involve exploiting an assumption in one part of the code to cause a vulnerability in what appears to be a completely separate part of the code, which is actually interdependent. Attackers and security researchers are very clever these days.

Is it easier to find and fix security bugs in Open Source/Free Software code rather than proprietary code? My personal feeling on that is "yes". Being able to examine the intent of the author by looking at the code and comments makes it much easier to see where the intent and the code don't match. That's where the security bugs live. Is Open Source/Free Software code safer than proprietary code? I'd say "yes" to that also. Looking at the number of vulnerabilities is misleading. A good 50 percent of our security bugs are found by our own internal code audits, or someone just looking over some code they're working on and saying to themselves "that looks funny..." Whenever we find something like that, we have to report it to the our vendor community and make a security announcement. It is painful doing this but as all our processes are open we don't have the luxury of silently fixing something and hoping no one will ever notice. Proprietary vendors do have that luxury; they would be insane not to take it, and I'm sure they do. Proprietary code vendors need all the help they can get competing with good Open Source/Free Software.

Samba is still written in C. I've examined the possibility of moving to C++, but to be honest that won't really help for the kind of security bugs that we seem to suffer from. In fact by hiding interdependencies in object classes it may make them more difficult to find. Also I'm not the final decision maker on that and many of the Samba Team hate C++ (Luddites that they are :-). Re-writing in a managed code language like Java or C#.NET would definitely help reduce our security problems but would reduce the performance that people find to be one of the compelling reasons to use Samba. It's no accident that none of the competing CIFS servers are written in managed code either. It would cost us six months to a year to re-write everything and get back to the same level of features, probably with reduced performance. We simply don't have the people or time to do that. For the moment we're stuck with C and we have to make the best of it. If someone is creating a competing Open Source/Free Software CIFS server I'd encourage you to do so in a managed code language as an experiment. If it works as well as I'd hope, we may throw our code away and join you :-).

In order to keep Samba relevant and current with modern practice, we've had to keep re-writing and re-designing core parts of the code. Our upcoming release, Samba 3.2, bears little relation to the original 1992 code and uses modern memory management techniques to avoid heap overflow, buffer overruns and memory leaks, much of which was invented by the Samba Team for internal use. The goal is to keep ourselves secure, as the Samba Team uses Samba for file services ourselves. This is all about scratching our own itch, and helping out everyone else as we do so.

I still feel bad about the mistakes I've made, but I try and redeem myself by using them as examples of how not to write code when I do interviews here at Google. If someone doesn't see the bug it always makes me feel better, but acts as a black mark against them in the interview score. After all, everyone we hire should be able to write code better than me :-). The code is out there to be studied; so learn from my mistakes and don't end up having to coordinate a security announcement. That's one learning experience it's better not to have.