Liberated Syndication: down, down, down

According to their website, "Liberated Syndication is a premiere media distribution service built from the ground up with DIY content creators in mind." Unfortunately, a DNS configuration error caused the site to be unreachable by customers for an extended period of time.

Here's the email they sent out to those unhappy customers:

---------- Forwarded message ---------- From: support@libsyn.com Date: Sat, 25 Aug 2007 14:49:51 -0400 Subject: Cannot find libsyn.com To: XXXXXXXX@XXXXX.com Dear Libsyn users-

I would first like to apologize for this email coming as late as it is. On thursday, 8/23 at approx 2:30pm EST there as a misconfiguration of DNS records sent out to the registrar which holds libsyn.com. This mistake quickly propagated around the internet in a matter of minutes, and despite our rapid response to correct the problem, there was a window where name servers around the globe stored and is currently saving the incorrect information. Initially we felt we had caught the error in time and that the effects would be minimal.

We posted to the normal channels- support.libsyn.com about the issue.

Several hours later, as we started to see reports of the outage (as users couldn't email us, cause the domain name was not resolving) we jumped back into the DNS issue and got on the phone w/ our registrar to see what they could do. Our registrar could not offer any solution to the problem other then waiting for servers around the world to correct themselves. They estimated it would take 24 hours or less. We are now approaching 48 hours and a portion of our users, and our users audience are still unable to resolve any .libsyn.com domains.

We continue to go back and forth with our registrar to see if there is ANYTHING we can do to speed up the global re-caching of our name servers to the proper settings. They keep telling us there is nothing anyone can do to force local, or regional DNS caches to expire.

There's a few layers of caching that occurs to make the Internet work, it seems. 1) source name servers- this is the ones we control. We can make changes and they are done instantly cause we control them

2) global/regional name servers- this is where our registrar pushes out the information we tell them. These we don't control and update slower then our source 3) local/ISP name servers- this is where your home computer gets its info from. They are run by the internet service providers (like Comcast or Verizon for example). These we don't control either and update at their own rate.

4) home computer- finally, your computer keeps its own cache so it doesn't have to hit your ISP's name servers everytime you hit a website you visit often. We definitely can't control these. We suggest a reboot of your computer to possibly kick-start your computer's dns cache. We apologizes for the repercussions of this error. We are doing everything in our power to bring full service back to all users.

Since the standard network communication channels http://support.libsyn.com and email (support@libsyn.com) were not able to be seen either, we have opened up some emergency channels which we will use in the future if there are issues regarding libsyn.com.

Wizzard.tv blog: http://www.wizzard.tv/blog Yahoo libsyn users group: http://tech.groups.yahoo.com/group/libsynusers

A few days ago, I described the need for careful testing to avoid making users want to curse you. In that spirit, here is free, unsolicited advice to anyone making DNS configuration changes: double-check your work before clicking that all-important CONFIRM button.

By the way, when your site is unavailable, especially to paying customers, the CEO should send out the dreaded email personally. An anonymous letter from support, as in this case, is pretty lame.

[Thank you to an anonymous reader for forwarding the email to me.]