Why Comcast is chasing DNS outages

If you're not a Comcast customer, you're probably blissfully unaware of the problems that Comcast customers have been experiencing the last few weeks.  If you are a Comcast customer, then like me, you've likely experienced serious downtime and you're probably wondering what's going on.  I've heard a few things through the grapevine and what I've heard hasn't made me very comfortable. 

I speculated on my blog that Comcast was getting early warning signals of impending disaster several weeks ago and that they ignored them.   What I've heard since is that Comcast essentially got caught with their pants down trying to support millions of customers on inadequate infrastructure.  They've been getting hit with recurring distributed denial of service attacks to their DNS infrastructure--such as it is.  The root problem seems to be that Comcast has a DNS architecture that consists of lots of scripts and some DNS software running on a couple of servers. 

Because they've got no management tools and little or no failover, when they get hit, they can't respond effectively.  They're essentially fighting a five alarm fire with a bucket brigade. As a consequence, they have had multiple, multi-million customer, multi-hour outages.
 
What's amazing is that such a huge service provider has been so neglectful of a core part of its architecture.  They ought to be using one or more reliable DNS service providers with rock-solid architectures, fail over, and management tools. 

I can sympathize with Comcast's position.  Excite@Home got themselves in this predicament and that, combined with an unsympathetic board, was their eventual undoing.   Ultimately, broadband companies have to face the fact that they differentiate on service and not much else.  If they're not investing in the infrastructure that makes that service rock-solid, then they're setting themselves up for longterm failure.  

CIOs ought to ask themselves what their DNS infrastructure looks like.  Are you running it yourself?  Should you be?  Outsourcing your DNS is not all that expensive in the grand scheme of things and is one less headache you've got.  Building a DNS infrastructure, not to mention training system administrators, to handle these kinds of attacks isn't easy and yet it may be the weakest link in your online presence. 

If you're a Comcast customer, what can you do in the meantime?  I was able to solve my problem because I had access to an alternate DNS provider.  You may not be as lucky.  I've also heard that setting up a local DNS cache (which many of the new consumer-grade routers do automatically) also helps.   A friend sent me these instructions for setting up a DNS cache on an OS X machine.   I'm sure similar instructions can be found for Linux and maybe even XP.