Remembering five years of vulnerability markets
Guest Editorial by David Endler
While reading the article, it hit me that it was exactly five years ago to the day I vocalized the idea of a vulnerability purchasing program. I was sitting in a tiny conference room at iDefense headquarters in Chantilly, Virginia. I was the director of their security intelligence division at the time. We had recently emerged from bankruptcy and we were having one of our first leadership meetings with the new CEO who had helped breathe new life into the company.
Each department head was going around the room with ideas for the 2.0 version of our startup. At the time, our main source of income was a subscription based newsletter called iAlert (hey, I didn't name it) that included vulnerability and cybercrime news items. When it came around to me, I proposed reforming our emergency notification procedures, increasing the size and scope of our research group, and paying external security researchers for vulnerabilities. I followed my idea with the observation that the group of people with the expertise required to discover a vulnerability was growing, and it was expanding outside of the elite security teams that historically had staked a claim on most vulnerability discoveries (eEye, Bindview, Xforce, etc.). I reasoned it made sense to reach out to this growing community in order to try and augment the vulnerability feed that iDefense provided at the time, giving us a competitive advantage over others in our market.
[ SEE: Auction sites opens up for vulnerabilities, exploits ]
At first there was silence. The CEO was the first to speak, "hmmm...not a bad idea. How would it work?" followed by an hour of intense group discussion from the entire room on a variety of details. How much would we pay? Would we give the vulnerabilities to the affected vendors? If so, how do we make sure the researcher stays quiet until the vendor produces a patch? How would we get the word out to the right researchers? The meeting ended with the task that I put a proposal together for final review.
I went off and put together the first draft for what eventually became the Vulnerability Contributor Program (VCP). Sunil James on my team helped refine a lot of the finer points, and eventually we announced it right after Black Hat that year on August 7th, 2002. As you can see from the initial prices that were offered, they pale in comparison to the amounts offered today by both iDefense and ZDI. Take for instance the $10,000 we recently paid to Dino Dai Zovi for his Quicktime
vulnerability discovery at the CanSecWest conference.
At the outset, I had a lot of concerns. Would security researchers take us seriously? What if the money wasn't enough? Would the industry as a whole condemn us to hell? Would product vendors even work with us once we tried to report bugs we purchased in their products? One thing I was certain of was that we needed a security research team dedicated to working with the VCP researchers. This led me to hiring Pedram Amini from my Alma Matter, Tulane. Over the years Pedram steadily grew the security research team at iDefense, until eventually coming to join me here at TippingPoint as our Manager of Security Research.
In the weeks that followed the VCP launch, there was the typical naysaying and failure predictions. Some felt we were offering too little money for what constituted many hours of a researcher's time. Others felt we were promoting hacking by creating a market for vulnerabilities. Some were just plain disgusted by the idea. Others made jokes at our expense, (some of which were amusing). These general viewpoints haven't changed much in five years. One of my favorite quotes from that time was from Wired News:
"When I initially heard that a company was preparing to offer financial rewards to security bug researchers, my first thought was that it would turn those exploit finders into prostitutes rushing around finding exploits to make a fast buck," said Marquis Grove of Security News Portal. "But as I thought further on the subject I came to the realization that over the years, everyone had been making money off the work of these researchers except the researchers."
I was pleasantly surprised by the number and quality of security researchers that approached us in the VCP's infancy. The biggest problem we faced though was that most of them didn't really trust us that much in the beginning; we were a small company and the program was still relatively unproven. It reminds me of the time I tried out eBay for the very first time. I bid on a small mug for a few dollars, just to see how the site worked. I figured, what's the worst that can happen; if I bought into a scam, so I'm out a few dollars. In much the same way, many of the researchers gave us fairly low severity vulnerabilities in the beginning just to test out the waters. We decided to buy them in order to build loyalty and trust, in hopes of getting the better stuff they were sitting on later. As a result, our first few public advisories were less than news worthy.
As time progressed however, our pool of researchers increased as did the quality of the vulnerabilities they submitted. Public security advisories resulting from the VCP also helped spread the word in the research community. Looking back now, we made a couple of missteps in the very beginning. But overall the program exceeded our expectations and our initial investments began to visibly pay off a year later.
One of the things that to this day frustrates me about the general iDefense business model was that we really couldn't do much for our customers besides warn them of an impending zero-day in our news feed. Many times we would include workarounds in the vulnerability feed descriptions, but in advance of a vendor patch, there wasn't much a customer could do enterprise-wide without disrupting business. Regarding disclosure, we also tended to tick off the software vendors fairly often. As soon as a researcher signed a contract assigning us rights to his vulnerability, we would pass it on promptly to the affected vendor. We would also send a description (minus exploit code) to all of our customers as well. This is where the problem with vendors would usually arise. This resulted in some of our more demanding mutual customers pestering the affected vendors for a patch, making relationships with those vendors hard to manage. Furthermore, there was always the fear of this valuable information leaking into the wrong hands. After all, it was delivered in full detail via a simple e-mail.
THE MICROSOFT RELATIONSHIP
I remember in one case, relations with Microsoft had broken down to the point where our responses back and forth had become pretty vitriolic. Regarding this specific vulnerability, Denial of Service in Microsoft ISA Server and MS Proxy, if you look down at the bottom of the advisory, you'll see an interesting disclosure timeline. Under the surface, it was a contentious back and forth disclosure effort. Iain Mulholland, who I actually count among my friends today, was the manager of Microsoft's Security Response Center at the time. Terri Forslof, who worked with him on the team, was also someone we communicated with frequently. Speaking for Microsoft at the time, both of them frowned upon our business model and would not agree to give us updates on the vulnerabilities we submitted to MSRC. Over the next few months, we went through several rounds of email banter, but never really reached any common ground. Until Las Vegas.
At the 2003 Black Hat Briefings, Microsoft was hosting their first ever researcher appreciation party. I ended up finally meeting Iain on the roof of Ghostbar and we argued loudly for a while, eventually gaining a mutual respect for each other and our respective disclosure stances. I'm constantly amazed how much can be accomplished over an open bar. I also met up with Terri Forslof later in the week, and we eventually ironed out ways in which Microsoft and iDefense could work together better. Over the next couple of years, Iain and Terri impressed on me a lot of the challenges that Microsoft faces with security response. Some of my ideas on disclosure changed as a result. I hope that I also impressed on them the challenges that security researchers often face while working with large bureaucratic software vendors.
After being at iDefense for five years, I had experienced most of the growing pains of a small struggling security startup and was ready to move on. I was proud of the things we had accomplished there, but my career had peaked and I was ready for a new challenge elsewhere. Being a glutton for punishment, I decided to leave for yet another security startup, TippingPoint. TippingPoint's bread and butter was developing Intrusion Prevention Systems (IPS). I signed on in 2004 to manage the department responsible for updating the new IPS vulnerability filters that get distributed each week to their customers.
About a year into my job at TippingPoint, I had been away from iDefense long enough to to reflect a little more objectively upon the vulnerability market we had propelled. In thinking about the next evolutionary step forward, the idea of a vulnerability auction actually came up. I knew that Greg Hoglund had first spearheaded the 0-bay idea, and almost made it happen, but had backed out at the 11th hour due to legal concerns. I started to wonder whether a company like TippingPoint could successfully get around those legal concerns and actually make it work.
The half-formed idea went something like this: Create an auction site that allowed a researcher to submit his vulnerability discovery. The site administrators would verify the vulnerability and make it available for bidding. Once the market determined the fair price of the given discovery, the affected software vendor would be given the first right of refusal to take the vulnerability off the auction table at the closed market price. Bidders would be limited to legitimate security vendors and perhaps some security service providers. The final winner would receive the vulnerability but be required under contract to disclose the issue to the affected vendor and give credit to the original researcher. The researcher of course would have to remain quiet about his finding until the vendor patched the bug. Ideally, the researcher would receive a higher pay out because the "market" decided how important his research was.
I say the idea was half-formed because as you can imagine, there are a lot of serious problems to overcome with this model. First of all, how to get product vendors to sign on to such a venture and not consider it extortion. How to get other security vendors to buy into this model from a competitor was also perplexing. Gleaning some value for TippingPoint customers was also a difficult proposition. Finally, the perception of auctioning off vulnerabilities to the highest bidder, regardless if they were "legitimate" buyers, seemed to touch a strong nerve with the few security people I approached with the idea. At the end of the day, I didn't see how this model could succeed, let alone fit into our corporate value system. This is not to say that WabiSabiLabi might not be successful. However, getting the security community to rally around the idea of an auction will take painstaking time and trust building.
After nixing the auction model idea, my team and I kept on coming back to the proposition of launching our own vulnerability purchasing program, eventually dubbed the Zero Day Initiative (ZDI). In order to create a program that was substantially different and better, we decided on a few unique choices early on to generate interest and build loyalty:
- A web portal for submitting vulnerabilities, tracking the status of submissions, and communicating with us.
- Use the acquired vulnerabilities to ONLY write IPS filters while the vendor was working on a patch. The IPS filter descriptions would remain vague enough so as not to leak the unpatched flaw. Otherwise we wouldn't release any details to our customers (no advance vulnerability notice) nor to the public - this was a huge departure from the iDefense model.
- A solid background verification process in order to make sure we knew exactly who our researchers were.
- Shortened turnaround time for vulnerability submission analysis.
- Easy and fast payment once a vulnerability is purchased.
- A group dedicated to reporting the vulnerabilities to the affected vendors.
I felt the last point in particular was crucial for making sure we reported these issues to vendors in a timely manner, and also received responses from them in just as timely a manner. When I was responsible for vendor disclosure in the iDefense days, I remember how grueling it was to keep up with the disclosure load as well as pinging unresponsive vendors. Right before I left iDefense, it would have taken up at least half of my day just doing that alone. While looking around for a capable manager to start this group, I was discussing the program we were going to launch with Terri (Iain had moved on to another group in MS). Her initial reaction, after having a heart attack, was relief that we weren't going to be sharing vulnerability details with anyone in advance of public disclosure. After a few conversations, I think she decided the best way to keep a close eye on us and make sure ZDI grew properly was to have a hand in it herself.
Long story short, Terri joined TippingPoint as our Manager of Security Response and has been the absolute perfect person for this role. Applying her experience of working with security researchers in the community, as well as her sympathy and expertise from the vendor world has enabled us to develop strong relationships with many of the vendors we contact regularly as a result of the ZDI.
Believe it or not, relations between TippingPoint and iDefense today are good. We've shared information on a number of occasions when necessary in order to weed out unethical researchers or researchers who have tried to double dip in both of our programs. The open lines of communication have been crucial in assuring that we don't end up with a frenzied bidding war when a researcher comes to both of us. There's plenty of security research to go around for purchasing, and at the end of the day, the researchers have ultimately benefited from another commercial player on the market.
It is apparent today that the vulnerability market is continuing to evolve. It will be interesting to see where we are in another five years. Will underground markets surge with an influx of cash from crime groups? Will vulnerability auction sites flourish due to a plethora of "anonymous" buyers? Will vendors finally put some skin in the game and start incentivising researchers more? Or will another small enterprising security group come up with another model that turns the market on its head?
Only time will tell.
* David Endler is the director of security research TippingPoint and also chairman and founder of the industry group Voice over IP Security Alliance (VOIPSA). More commentary from Endler can be found on TippingPoint's DVLabs blog.