We use URL shortening services so often these days we're seldom even aware of them. Until, that is, one of them fails. Today, one of the most popular of these, Bitly, stopped working for some users.
You may think of URL shorteners as something individuals use for shortening long web URLs. This is true, but companies also use them to promote their webpages on social networks. In addition, Bitly enables business marketing offices to track who's clicking through on their links. While a Bitly outage is an annoyance for someone sharing LOL cats pics, it's a major problem for businesses.
Eventually, it became clear that the problem was only affecting Comcast customers. Bitly then tried to "put mitigations in place to work around the problem for users on Comcast."
Unfortunately, these didn't work out. "We have had to temporarily remove the mitigations that were put in place to work around the connectivity problem for Comcast customers."
By noon Eastern time, Bitly finally reported: "Mitigation is fully in place and we expect no further impact to users. We will continue to monitor and provide updates should any further developments arise."
So, what happened?
It turns out Comcast had turned off redirects on the "bit.ly" domain. Bitly CTO Rob Platzer told ZDNet:
Last night Comcast's security team blocked the IPs for Bitly due to reported phishing attacks using the domain. We have blocked the responsible accounts and links and are working with Comcast to get bit.ly unblocked as soon as possible. In the meantime we have moved over to our backup network to mitigate the outage.
In an additional statement, Platzer said, "We have multiple systems both automated and manual to monitor the Bitly platform for abuse and flag malicious links. These include real time integrations with industry standard feeds of malicious domains, sites, users, IPs and other assets. On this occasion, unfortunately, Comcast automatically detected spam and accidentally shut down redirects on the 'bit.ly' domain before engaging us."