The Harsh Reality of Suburban Broadband, Part Deux
Goddamit! Why have I been offline for so long?
As I explained in a previous article last summer about my broadband situation, my employer, like many large technology services companies, has elected to classify my work situation as "Home-Based", in that when I am not at a customer site, I'm working from home. So for connectivity to corporate email, our Intranet and Instant Messaging system, my link to the mother ship is entirely dependent on the reliability of my connectivity through Optimum Online, Cablevision's cable broadband service.
Click on the "Read the rest of this entry" link below for more
Service interruptions with residential broadband are not uncommon, in particular with cable and DSL providers. It's not unusual that once or twice a month, I'll have to re-boot my cable modem or have a 2 or 4 hour service interruption. I don't usually notice it though, as I spend a great deal of time traveling and my wife is usually the one complaining when it goes down. If she can't get onto Rummikub.com, GMail or FaceBook, I'm usually the first to get a phone call.
"Honey, are you downloading something really huge?"
"No, I'm not. I noticed all my connections dropped, are you doing something?"
The Surfboard was lit up like a Christmas tree. I rebooted it twice, as well as my router/firewall. Same problem. Modem lights going crazy but the PCs have no apps running. DVRs aren't downloading anything over On Demand either. I'm thinking DOS attack, or some script kiddie in the neighborhood is screwing around. So I call up Cablevision.
Also Read: The Harsh Reality of Suburban Broadband (June 2008)
First, I get to the voice response system. Press 1 for English. "Empuje el numero dos para espanol". Press 1 for Sales and Appointments, Press 2 for Billing. No option for technical support, unless you hit zero a bunch of times causing to system to say "I didn't understand that, forwarding you to a representative." A rep picks up after about 40 seconds, where she then proceeds to look up my account number and service history. Then when I finally get to tell the representative I'm having a technical problem, I get transferred to "Mike".
I explain to Mike I'm getting a ton of incoming traffic and I can't connect to anything, this despite my router having leased a WAN IP address from Cablevision's servers. Mike tells me from his remote diagnostics that I'm dropping a ton of packets, and suspects the modem is bad, and to get it swapped out the in morning from a local Optimum/Cablevision retail location.
Thursday morning I wake up and head right over to the local Optimum store. Optimum swaps out my presumably dead Surfboard for a Scientific Atlanta, a smaller modem with the equivalent build quality of a Chinese-made child's toy, and a date of manufacture of April of 2004. I don't complain about this because next Friday, I'm being upgraded to Optimum "Ultra" service, their competitive broadband offering to Verizon's FiOS.
I happened to find out about Ultra when I asked Mike if Optimum had rolled out DOCSIS 3.0yet, and he said that they literally just got it up a few weeks earlier. At $99 per month (as well as a $300 equipment fee for the new channel-bonding DOCSIS 3 modem and an on-site tech visit) for 100Mbps downstream and 15MPBS upstream, it sounds like a hell of a deal compared to my current $65 per month Optimum Boost service, which is 20Mbps down/5Mbps up.
I've been doing a lot more Internet streaming intensive stuff such as Netflix instant view with Roku and DirecTV On Demand, as well as a lot more usage of my Slingbox when I'm away, so the increased bandwidth should be really nice. I had been pining for Verizon FiOS, but after speaking to the mayor of my town about the prospects of it happening anytime soon, he told us we are on the bottom of Verizon's list and I'll be old and grey by the time it ever happens. So Ultra should be fantastic if it lives up to its promises.
In any case, I bring the replacement modem home and hook it up. It lights up like a Christmas tree, just like the Surfboard did. I yank my router's Ethernet and do a direct CAT-5 bypass from my laptop, pull up Wireshark, and begin capturing on my primary interface, which spits out a ton of errors about dropped packets and malformed TCP requests. GODDAMIT!!!!!
I call up the tech support line again. It's 11AM. I call back the same support phone number, but now it takes a bit longer to get through to the technician. I explain to him, again, the packet situation and that the modem has been replaced.
"Well Sir, it does look like there is some sort of outage situation going on, but the system doesn't have any updates about it logged since 8:05AM."
"So you can't definitively tell me if people are working on it or what exactly is wrong?"
"No Sir, I cannot."
I call back in at 1PM, effectively having the same conversation. Frustrated and angry, I hang up the phone. My Internet access has now been out for 14 hours. My entire business day is on the verge of being completely shot. Thankfully, my BlackBerry works, I'm able to tether it to my laptop for some limited 3G email and web browsing, and I haven't had any urgent company messages or issues that need to be addressed.
At this point, if I were the average Joe, I would have had no recourse. He would have waited, and waited, and waited, and maybe at some point that day, or the next, or perhaps the day after, his service would have come back. Maybe. But I'm not the average Joe. I'm a columnist for a high-profile tech blogging site, and when I bitch and complain, people tend to listen. So I do the only thing a mad-as-hell tech journalist does when deprived of his broadband. I jump on Twitter with my BlackBerry.
Not surprisingly, right after my first complaint that the modem swap didn't do any good, I have some direct Twitter messages Jim Maiella, Cablevision's Vice President of Media Relations waiting for me. Before you can say "Don't incur the wrath of the fat angry Jewish dude from New Jersey" I have the regional Director of Network Engineering on the phone, and two Cable Guys at my house and casing the neighborhood for several hours chasing what appears to be a systemic problem that is now affecting a large number of customers, not just my "Node".
Nevertheless. I get my modem swapped out AGAIN, for a Motorola surfboard just like the one I first had, because the on-site tech tells me the Scientific Atlanta I've been given is a piece of crap.
Predictably, the new-new Motorola lights up like a Christmas tree, and I show him the Wireshark readouts with the red errors and warning signs about packet loss and malformed requests. I also show him that they've got a CISCO router or switch going nuts on Cablevision's internal network, sending out a broadcast storm that is effectively causing a denial of service. Not alarmed by this, he walks outside and attaches a portable network analyzer to my coax drop at the house. 60-70 percent packet loss.
The Cable Guy leaves and comes back a few hours later -- 8:30PM, telling me that my service has been restored. I've experienced an outage of approximately 22 hours.
The cable guy informs me that the problem was "On the back end, related to switch port configuration." OH REALLY? I HAD NO IDEA!!!!
I think there are a couple of things that need to be added to the lessons learned pile here, and that is that I don't think broadband customers should need to be authors of high-profile tech blogs in order to get prompt customer service. If there's an outage, some sort of update as to the nature of the problem would be nice.
Additionally, if you're running a large residential broadband provider, it's probably a good idea to have some sort of low level network monitoring of all your manageable devices put in place so that you can trigger and send alerts of exactly what is going wrong to the right people who can fix the problems. Having cable guys chase down endpoints at customer premises when the problem is on the back-end is a huge time waster.
Has anyone else experienced a significant downtime or lousy customer service with their residential broadband provider? Talk Back and Let Me Know.