Why Comcast is chasing DNS outages

Why Comcast is chasing DNS outages

Summary: If you're not a Comcast customer, you're probably blissfully unaware of the problems that Comcast customers have been experiencing the last few weeks.  If you are a Comcast customer, then like me, you've likely experienced serious downtime and you're probably wondering what's going on.

SHARE:
TOPICS: Networking
15

If you're not a Comcast customer, you're probably blissfully unaware of the problems that Comcast customers have been experiencing the last few weeks.  If you are a Comcast customer, then like me, you've likely experienced serious downtime and you're probably wondering what's going on.  I've heard a few things through the grapevine and what I've heard hasn't made me very comfortable. 

I speculated on my blog that Comcast was getting early warning signals of impending disaster several weeks ago and that they ignored them.   What I've heard since is that Comcast essentially got caught with their pants down trying to support millions of customers on inadequate infrastructure.  They've been getting hit with recurring distributed denial of service attacks to their DNS infrastructure--such as it is.  The root problem seems to be that Comcast has a DNS architecture that consists of lots of scripts and some DNS software running on a couple of servers. 

Because they've got no management tools and little or no failover, when they get hit, they can't respond effectively.  They're essentially fighting a five alarm fire with a bucket brigade. As a consequence, they have had multiple, multi-million customer, multi-hour outages.
 
What's amazing is that such a huge service provider has been so neglectful of a core part of its architecture.  They ought to be using one or more reliable DNS service providers with rock-solid architectures, fail over, and management tools. 

I can sympathize with Comcast's position.  Excite@Home got themselves in this predicament and that, combined with an unsympathetic board, was their eventual undoing.   Ultimately, broadband companies have to face the fact that they differentiate on service and not much else.  If they're not investing in the infrastructure that makes that service rock-solid, then they're setting themselves up for longterm failure.  

CIOs ought to ask themselves what their DNS infrastructure looks like.  Are you running it yourself?  Should you be?  Outsourcing your DNS is not all that expensive in the grand scheme of things and is one less headache you've got.  Building a DNS infrastructure, not to mention training system administrators, to handle these kinds of attacks isn't easy and yet it may be the weakest link in your online presence. 

If you're a Comcast customer, what can you do in the meantime?  I was able to solve my problem because I had access to an alternate DNS provider.  You may not be as lucky.  I've also heard that setting up a local DNS cache (which many of the new consumer-grade routers do automatically) also helps.   A friend sent me these instructions for setting up a DNS cache on an OS X machine.   I'm sure similar instructions can be found for Linux and maybe even XP.
 

Topic: Networking

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

15 comments
Log in or register to join the discussion
  • Comcast & DNS outages

    I knew something was amiss.

    Try calling Comcast during one of these outages and you will be even more angry.

    I was put on hold for 20 minutes and finally gave up.
    Lorenzo1950
    • Comcast outages

      I've been a Comcast customer since they bought @Home... Needless to say if SBC would get off their rear-ends and upgrade the neighborhood's RT, I'd be on DSL in a heartbeat.

      My strategy has been to nickel and dime Comcast everytime there is an outage. I am the customer they want least to hear from because that usually means there is a problem they can't attribute to Customer premises equipment...

      They don't bother telling you when there are planned outages, nor do they tell you when the outage is expected to be "resolved."

      So, whenever there is an unplanned, uncommunicated outage, I complain until I get credit on my monthly bill. If every affected customer did that, it would hurt enough for the bean-counters and execs calling the shots to actually do something. They'll see what it's like to have a $0 revenue day like I have as a remote IT consultant.
      jhealer@...
  • Message has been deleted.

    ip_fresh@...
  • Message has been deleted.

    ip_fresh@...
  • Ignoring "ripples" oftimes leads to cascade failures

    Quite agrees, does seem Comcast has not made the effort to stay ahead of their infrastructure and basic "housekeeping" to avert these outages.

    Sad to say, is often the case it takes a cascade failure to get draw attention to the facts, put resources (talented tech support, bandwidth capacity planning and a "fast response" mentality) where they're required.

    Ignoring (or worse yet not being aware of) the small rippes in the pond, oftimes, leads to a cascade failure since they tend to multiply over time and magnify quickly.

    There's no genuine excuse for a company like Comcast being "caught unaware" and not investing in the infrastructure (human, technology and capacity) to address these scenarios. Afterall, they're not exactly the "new kids on the block" when it comes to being a communications service provider.

    Just my views.
    slaldrich
  • failure to perform

    from excite@home to comcast the wiring of our 400 unit condo
    complex has not been maintained. original wires are above
    ground and heavy rain also causes problems. in six months
    there has not been a 30 day period of uninterrupted service. one
    outage lasted 7 days. previous calls for help were not addressed
    until the condo association started to get tough.
    meadfair
    • failure to perform

      Just for the record, Excite@Home never had responsibility for lines in your condo or any other consumer property. E@H was the national backbone that linked the various cable providers together. The cable providers were always resonsible for the local plant.
      windley
  • Comcast Outages

    ATT Broadband used to purchase good DNS software but they have a different management crew now compared to Comcast. For the most part, ATT folks knew the value of logical and physical networks- but look at what happened to ATT. Use the term "logical network" in front of a Comcast exec and watch their face make contortions like that Keystone beer commercial from last year. It's not pretty.

    The difference is apt when you see a Telecom vs cable or satellite ops center. The satellite and cable companies have tiny little dark rooms with 100s of 5" TV screens. They eyeball the screens for errors after they have occurred. Imagine trying to explain DNS or SNMP to folks monitoring 5" TVs. Then try to explain IPTV to Verizon, Comcast or GE aka NBC.

    Video and cable providers have the RF perspective so they spend millions on physical infrastructure and neglect minor details like DNS. They don't make a profit linked directly to DNS so why should they bother. Massive outages haven?t stopped them from profiting millions. Why should Comcast care?

    The big question is what type of credit the Comcast customers demand on their monthly bills. If users consistently force the issue in terms Comcast understands, DNS might improve for the end user.
    mijuguete
  • love my comcast

    super fast, they are always nice and efficent.. maybe they just suck in some areas more than others.

    As far as these recent DNS problems, i noticed them and was without being able to resolve domain names for a total of 20 minutes between all the reported outages... i havent really seen an issue. most of the time on on my comcast though im not using domain names so it didnt matter much to me.
    doh123
  • Can't afford DNS but will buy Disney??

    Glad I switched to Earthlink DSL from the lame service I was getting from Comcast. Sure, they boast about "upgrading" your service but my horrible pings were always due to their failure to upgrade overloaded routers that reduced my $60+ a month connection to crap. Now, I pay half that and get better service. Good Riddens to another badly run (cable) company that is more concerned about fleecing the "customer" than providing a service!

    IMO, until there is two or more choices for the exact same service connection, the government needs to step in and control these monopolies especially in regards to their pricing. It's insane what these cable companies are charging especially when one can consider buying another company the size of Disney.
    Teyecoon
  • No Comcast outages I know of ...but...

    I checked with _a few_ Comcast people I know and heard of no outrageous outages. They obviously occurred, but occurred at an "ideal" time for an outage -- when most business users aren't on the air.

    But the unpleasant duty of fronting up to informing customers of software shortcomings or abridged service is a woefully underdeveloped art in every software and telecomm company whose products/services I have used. Some do it well in fits and starts, but inconsistently, depending upon the guts of the relevant VP or prez.
    Pancracio
  • They blamed me

    Starting early March I began having outages and slowdowns. Reporting outages resulted in endless "troubleshooting" and blaming my equipment. Eventually I refused to troubleshoot, and demanded a technician. They offered to send one but it he didn't find a problem at Comcast's end, I'd be billed at least $40 up. After telling them where to get off, they agreed to send one free. Result: 1. the line into my home was shredded, 2. the original (TV) line inside my home should be replaced, and someone had left an inappropriate filter on it, and 3. the signal from outside wasn't strong enough. Problem continues. I got almost a months credit, little enough as I work on the internet and it's cost me money!!! Problem continues on and off.
    Bess
    • Me Too!

      I have had at least five tech visits over the last two months. Every bit of cable has been replaced and until the last guy came out, no one would believe me that I was sure it was a problem in their network. He did give me the number of his supervisor to call, but I haven't been home during his working hours and experienced an outage. I'll give him a call and get some of this info to him.

      Does anyone have contact numbers or emails we can use to complain?

      At this point, I'm just happy to have found out that my feeling that it was DNS servers ahs been confirmed!

      Andy
      ahlittle
    • Dust off the typewriter

      Write a friendly letter to Comcast, if they don't solve their problem in 30 days, write to the BBB, the FCC and/or your congressman.
      Oh, I forgot to tell you, in the letter you write to comcast, tell them you'll also write to these friendly people if they don't solve your problem
      SantiagoCrespo
  • Things you can do when Comcast denys problem.

    As you write your letter to the FCC, Congressman, etc. Consider providing them proof that Comcast has failed to maintain service.

    1. Offer proof: provide them with trace route reports proving a problem exists. I have a small list of a few websites that originate in different regions of the USA and I use the command 'tracert' to generate a report of response times to prove something's wrong with their connectivity and not with my equipment.
    You could even gather this information when service is running normally so that you can show the difference.

    For reference, response times under 50 millseconds are normal. Greater than that are abnormal. However, please note that Comcast uses 100 milliseconds as their criterion for normal.

    2. While comcast service is problematic, you can take action to work around their current problem. Alter your home Router/Gateway settings to point to another DNS server. I use the free DNS service from opendns.com. It's simple. They even give you instructions on how to make the changes for just about every router/gateway there is. Note that this doesn't prevent poor connectivity but is does give you access when comcast's DNS server is under attack. Plus opendns.com protects your computer from other threats by blocking/filtering malicious content.
    jameskel@...