Vulnerability disclosure gone awry: Understanding the DNS debacle

Vulnerability disclosure gone awry: Understanding the DNS debacle

Summary: On July 7, the day before the release of the patch for the now infamous DNS design flaw, hacker Dan Kaminsky (with the help of Black Hat conference organizers) invited reporters to a press conference to "discuss the massive multivendor patch being released this Tuesday.""A synchronized release of this magnitude has not happened before," read the invitation sent to the Black Hat conference press list.


Vulnerability disclosure gone awryOn July 7, the day before the release of the patch for the now infamous DNS design flaw, hacker Dan Kaminsky (with the help of Black Hat conference organizers) invited reporters to a press conference to "discuss the massive multivendor patch being released this Tuesday."

"A synchronized release of this magnitude has not happened before," read the invitation sent to the Black Hat conference press list.

By the time the patch was released, Kaminsky had briefed influential bloggers, recorded podcasts, scored a Wall Street Journal hit, celebrated an appearance on the front page of the BBC and won respect from his peers for coordinating such a massive cross-vendor patching effort.

It was a patching initiative that required six months of secrecy when countless security folks had to be kept in the loop.  Potential patching hiccups had to be sorted out, important advisories/mitigations had to be prepared, DNS forwarding instructions had to be ready.  A near impossible task, executed to perfection.

But, as Kaminsky admitted up front -- and would soon find out -- he made a major mistake of ignoring his peers in the hacker community, an intensely curious group prone to jealousies and stealing each other's thunder.

In the days following the release of the patch, Kaminsky declined to provide technical details, insisting that affected vendors and end users needed at least 30 days to properly test and deploy the fix.   Funny enough, the self-imposed 30-day deadline would end at the Black Hat conference where, at 11:15 a.m., Kaminsky would take to the stage and bask in the glory of his discovery.

Thomas Ptacek (right), principal of Matasano Security, was the first to call BS on the secrecy.   Kaminsky immediately arranged a private conference call to spill the beans.   Dino Dai Zovi, another researcher with hacker cred, was included.   After the call, both Ptacek and Dai Zovi confirmed this was something super-serious that required immediate attention.

It was not enough.  Monitoring the security mailing lists (Daily Dave, Full Disclosure, etc.), you could sense the backlash growing.  Kaminsky's request for a moratorium on public speculation -- he even promised a Black Hat co-appearance for those who figured out the bug but maintained secrecy -- did not sit will with everyone, including Ptacek.

Brand-name researchers started to grumble about the "cabal" approach to disclosure, openly venting that non-speculation and non-disclosure even after patch release were tantamount to being irresponsible.

Paul VixiePaul Vixie, of BIND fame, joined Kaminsky in pleading for the embargo but it was clear that public speculation would eventually emerge.   It was only a matter of time before someone smart figured out how to forge and poison DNS lookups.

Halvar Flake (right), a reverse engineering guru who was among those arguing for public disclosure, published a guess/hypothesis that (almost) nailed the bug.

Ptacek's Matasano followed up with a de-facto confirmation that filled in the missing pieces (the blog entry has since been pulled but the deed was done), forcing Kaminsky to acknowledge that his Black Hat thunder was stolen.   Ptacek has since apologised but there are so many ruffled feathers, it's hard to imagine things being the same in the land of trust/coordination/disclosure.

There's a long list of researchers who argue that Kaminsky's embargo was nothing but hype for the Black Hat conference.  Kaminsky admits to being a media hacker and his pre-patch press conference and appearance on subsequent Black Hat marketing webcasts have done little to quell those concerns.

However, throughout this episode, I always got the sense that Kaminsky was genuine about wanting to give people adequate time to test and deploy the patch before things got ugly.  Kaminsky has earned the right to be trusted on the severity of DNS-related issues so it's sad that this debacle occured on his watch.

A lot of it was his own doing but, in the final analysis, maybe he deserved better.

There's a lesson in here somewhere for those who try to figure out the politics and drama surrounding vulnerability disclosure.

Topics: Security, Browser, Networking

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • I think he deserved better for sure

    You know, Dan does manipulate the media well, but I'll tell you this, he's a stand-up guy. Did he try to drum up the press a bit? Sure, why not? Did he choose his method of disclosure and call for no attempts to figure his research out for media frenzy? I say no way.

    This was a huge bug. When I saw the details of just how simple it was, I realized it was what I expected... something that exploit code could be out for within a number of hours, not days.

    I think Dan did the best he could to protect people for as long as possible, and no matter your thought on that, that is respectable.

    For the record, I wish it would've worked out differently for everyone involved... I consider them all great guys. Also, I'll still be going to Dan's talk. Halvar commented on this on his blog, saying to now not go to Dan's talk would be like going to see a renowned physicist and some dude on the street tells you what he'll say... would you still go? Of course. Now, Halvar's no dude on the street, but Dan's talk will still be 100% entertainment, because, if for nothing else, Dan is a great speaker and this is a huge flaw.

  • to be honest,

    I think this tempest in a teapot underlines that there is very much maturing which needs to come.

    I thank you for linking references, Ryan. Then it becomes clearer just how envious and selfish the thinking and actions were by some. Apologies after the fact are welcome, but you really have to consider the lack of consideration of consequences.

    It would be very hard to consider using the services of Ptacek or Flake in any future way.

    Where was their consideration of the vast numbers of persons depending now one way or another on the internet?

    And I feel this episode underlines how the 'disclosure culture' is just wrong, and based mostly on egotism. We should reward differently, and I think Kaminsky was trying to do that - including all who can contribute.

    It's much better than a 'chicken' contest to see which one individual can be momentary top dog.

    Because in a moment, that's gone, while community rewards and goes forwards.

    Narr Vi
    Narr vi
  • What was the point I wonder...

    This is tough...but there is something to the argument that point to the media-hype aspects to how Dan managed this issue.

    BUT, really I have to believe that the idea of handling the way he did and to announce the bug and fix at the show would have been a good way to highlight how we in the security community are working together to make it better.

    On that note I am kind of bummed that this played out as it did. Had everyone acted on cue, I think it could have been a good thing for us all to say, "Hey, were are working together here to make this better and contribute". As usual, egos got in the way.

    Kind of sad actually. What was the motivation? Was it it to make it contribute or to steal another's contibution and attempt to fix for the greater good for a substitute agenda to diminish Dan's efforts?
  • What a pointless article

    People are not perfect. But a definite service was performed. Case closed.
    The Rationalist
  • RE: Vulnerability disclosure gone awry: Understanding the DNS debacle

    Kaminsky did the right thing. Vixie too. Internet is too critical to allow ego to trump security. this is not to say that vulnerabilities should be hidden. They must be fixed. But in a responsible way.
  • RE: Vulnerability disclosure gone awry: Understanding the DNS debacle

    Ryan Naraine needs to move over to the Hollywood Reporter,
    where his dirt-mongerning instincts will be more appreciated.
    • Where did that stem from?

      You can't say something like that and not point to why...

      • Ya know?

        The biggest hoo-hah I've seen throughout this whole thing has been generated by some of the people who comment on blogs, and maybe some of the non-tech media coverage.

        Both the "wait for it" and "disclose now" sides have equally valid reasons for their positions. For the onlooker, it's like trying to choose between the greater of two goods, without enough information. (Aside from those who just like to rant.)
  • This crazy world.

    A people that kill other and cause terror under the civilian population is a terrorist but (at the same time) a people that kill other and cause terror under the civilian population is a hero.

    And now, a hacker, and most directly a black-hat is a savior and a pro-security?.

  • Try telling a bank you can break their systems

    Late in 2006 I figured out whay so many of my friends were getting locked out of their online banking.

    So being a good guy, I tried to contcat the bank to tell them about it. This took months, and even after I wrote to them I was told (to paraphrase) "don't worry your pretty little head about it".

    It is not like I am a lunatic off the street. As well as working in IT for a long time (I go back to 5 hole paper tape, and using switches and LEDs to load programs), I have also managed R&D projects on new techniques for smartcards.

    After one year, with the flaw still not addressed, I tried people who monitor the banks, and finally the bank called me up. Not that they would discuss anything.

    However, some very subtle changes have appeared since which makes me think that they may have alleviated it.

    The fundamental flaw still exists and means that it is impossible to fully guard against a denial of service attack. I believe their actions prevent it happening from a few PCs, but would not stop a botnet taking the bank down.

    Thus, a bit like Dan, I am in a "no-win" situation - do I keep quiet and hope the bank will do something; but there will be no comfort when they inevitably get attacked in saying "I told you so". Or do I go public, which could jeopardise the bank, esepcially when so many banks are fragile at present?
    • Re:Try telling a bank.....

      I'd try writing to the CEO.

      Nobody likes being told they screwed up but it happens. The best way to measure a company's honesty is to see how they deal with it. Sometimes the worst person to inform of a problem is the guy in charge. Dept. heads stand the most to lose so they don't want to rock the boat. The CEO on the other doesn't want to look foolish either and should welcome comments of your nature.
    • To Tell of Not to Tell: Human Nature v. the Savior Instinct

      The would-be savior desires to save the world, but what if the world or the individual is so prideful that they refuse to recognize danger or refuse to be saved. The would-be savior, one who also posseses power, because of his own pride may feel a desire to use his power to lash back at being snubbed. In other words, human nature works to turn a savior into a destroyer. The answer to the dilemma - the same dilemma that confronted Hippocrates - is to to do what you can, allow people their freedom, and do no harm.
      • Great analogy (Hippocrates, I mean) !

        And like all members of helping specialties since Hippocrates, Kaminsky found himself in a no-win situation - accused of having an agenda for his actions no matter what he did.

        To paraphrase one of the greats of baseball, Kaminsky might be forgiven for thinking he shoulda stood in bed when he did what he thought was right to minimize the fallout from this particular vulnerability warning. He should have been given credit for scheduling formal release of information about this particular vulnerability for a time when people would have been listening to him anyway.
  • RE: Vulnerability disclosure gone awry: Understanding the DNS debacle

    Everyone in that article is wearing a black shirt.

    I think this could be part of the problem. Hackers need to steal thunder, but also, deep down, they are annoyed that everyone else dresses exactly like them.
  • I'm a simple guy... w/ simple needs...

    Just a side note...

    I'm not an IT Guru. I can barely make a page using HTML.

    I know that I 'need' and IP Address and the DNS has to work for me to be online... but that's about it.

    I'm sure You IT folks are aware of this (and I've seen blogs and such regarding this), but what I guess I'm getting at is...

    While I do take some measures to understand how these technical things work, I just want to be able to get online, read email, post to my own meager website...


    Thanks for All the Work You IT folks Do!
  • The important thing...

    The important thing is that the damn thing got fixed before it caused any disasters. Kaminsky should take pride in the fact that he made this happen, and his peers should be proud that one of their own saved the day. White hat, black hat or hatless, everybody needs a functioning Internet.
  • Actually it all might be for the best

    The more publicity there is about the problem, the more likely that real-world DNS servers get patched now. I had downloaded it when the news first broke, but installing it wasn't high on my list until this recent kerfuffle. I figured I had until Black Hat.
  • RE: Vulnerability disclosure gone awry: Understanding the DNS debacle

    This looks like a bad soap opera. Again no matter what position you take about the disclosure of the DNS vulnerability, you must patch your DNS server and client ASAP since now this vulnerability is out in the wild and cannot be recalled.
  • Understanding the DNS debacle

    Dan Kaminsky is astute enough to realise that he might upset others by maintaining the secrecy about the source code, even after the patch's release.

    He is also astute enough to know the severity of possible damage to the internet infrastructure had the vulnerability been exploited anywhere 'out in the wild'

    FULL MARKS AND TOP CREDIT to Dan and the DNS server people for completely keeping the lid on a potentially VERY difficult situation.