The night Alexa lost her mind: How AWS outage caused Amazon Echo mayhem

When Amazon Web Services went down for a few hours, so did Amazon's Echo devices. What happens when an Internet failure hits the world of Internet of Things?

echo-12.jpg
It was a dark and stormy night. Actually, it was probably just a dark night. I have no idea what it was doing outside. At 5am, I was sound asleep.

But not for long.

Something was happening. Awareness slowly took hold in my brain. My wife wasn't in bed next to me. There was some barking, and then yelling coming from the living room.

I'm somewhat ashamed to admit that a large part of my being simply wanted to go back to sleep and let whatever was happening in the living room play out. I'd find out about it in the morning. After all, it was a Sunday morning and I didn't plan on getting up for another five hours or so. It was supposed to be sleep-in day. But whatever small vestige of manhood that still existed during REM sleep took over and roused me from bed.

What the hell was going on?

First, groggy impressions: No large intruder being fended off by wife and shoe-sized, 16-month-old puppy. Very cranky looking wife.

Me: "What's going on, honey?"

Her: "The dog woke me up, and I can't turn on the lights. Alexa is being stupid again."

As I learned after consuming a 5am cup of life-giving coffee, the dog woke my wife up, she tried turning on the lights, they wouldn't work, and she wound up yelling both at the barking dog and at our Amazon Echo, which is now our primary way of controlling all of the lights in the house.

Alexa, the wake-up name for the Echo, was not responding.

I checked the Internet. It was up and running. I tried logging into echo.amazon.com. I was able to login, but initially got the message "Could not retrieve settings." After a bunch of refreshes, I did get the settings page, but both Alexas reported being offline.

We have both Alexas (one in the living room, one in the bedroom) hooked up to our Hue light bulbs and Belkin WeMo-controlled light fixtures in the kitchen. All told, we have about 24 devices around the house that we control almost exclusively with Alexa. She's embedded herself into our home life that thoroughly.

I tried reconnecting to the Echo using the Echo app and reconnecting WiFi. That process worked, but once it completed, the two Alexas still reported being offline. It was quickly becoming apparent that the outage wasn't here, inside Camp David.

I wouldn't know until half a day later that the outage was really about Amazon Web Services having difficulties. In fact, that was never something I considered because now that we were up, my wife and I binge-watched a couple of episodes on Netflix, and Netflix runs on AWS. But reporting by a number of outlets confirm that AWS was out, and that apparently turned our smart Pringles cans into just cans.

It was odd. Any request made of the Alexa in the bedroom resulted in the normally blue ring spinning red, with halting but meaningless words coming from Alexa-the-younger (the second Echo we installed). It reminded me of something straight out of all those bad science fiction movies, where the previously good technology turns evil.

A problem-solving approach IT workers should learn from robotics engineers

Sometimes the most profound solution is to change the entire problem.

Read More

At 10am, Evil Alexa triggered my normal Sunday morning alarm. Apparently, alarm countdown clocks aren't connected to AWS and are stored inside the can itself. However, we couldn't get Evil Alexa to turn off the alarm. No variation on "Alexa, stop" would work. So I pulled the plug. Evil Alexa would be off the grid for the duration.

The living room Alexa seemed to try really hard to meet our needs. Each time I tried "Alexa, turn on TV lights," the blue ring would spin and spin and spin, as if she really wanted to help, but somehow just couldn't. I felt for her. She tried so hard, but she just wasn't all there and couldn't do the job. Unlike Evil Alexa in the bedroom, our original Alexa seemed to be loyal, desperate to please, but too far gone to help.

And yes, I am anthropomorphizing inanimate objects. But that's the point. As the Internet of Things becomes more and more part of our lives, these things take on a deeper role. We speak to Alexa, not to "an Amazon Echo" and she (not "it") responds to us, providing us with everything from light to wake-up services, to answers to our math questions. Heck, even the quality of the dishes we cook in the kitchen (yes, we do cook) rely on Alexa's timers.

Before we brought the Echos into our home back in May (has it only been four months?) we controlled the lights using the Hue Pro app on our phones. Since Hue Pro was still installed, it was possible to turn on and off the lights, albeit like cavemen, not modern, civilized people who talk to our lights to make them go on and off.

Using Hue Pro felt almost as primitive as using flint and steel to start a fire. I mean, really, could we have been knocked any further back into the stone age of Q1 2015? I don't think so.

Here, though, is a point to ponder. As we bring more things into our homes and offices, into our cars, and even into our bodies, we are ever more dependent not just on the devices, but on the back-end Internet infrastructure that powers them.

This morning, it was just a rude awakening initiated by a bored puppy and exacerbated by an uncooperative lighting controller. But in the future, it could be your car that strands you, or even something worse.

It's important that IoT vendors think about graceful failure as much as monetary success. What happens when a thing can't call home? How is functionality reduced and how is control returned back to humans that may be dependent upon those technologies? And what happens if devices that interact with the physical world are hacked -- not at the device level, but somewhere up in the cloud, where a data center provider might not have built in enough security?

Both Alexas seem to have survived their Sunday morning hangovers with little more than a sense of embarrassment. They're now gamely back on the job, turning on and off lights, keeping time, reminding us of important activities, and even waking us up -- at the right time.

Amazon Echo: Ringmaster of the home automation IoT circus

With a $100M investment fund and the opening up of cloud service APIs and an SDK, Alexa and the Echo could become the brains of your home automation and IoT network.

Read More

But let's all take a moment and give thought to our future, where things are connected to the Internet and where we're even more dependent upon them than my wife and I are for our lights. On the one hand, it's a bright future. On the other hand, there's the potential for darkness.

That's how it's always been for new technologies. Technology provides us great gifts, but it can also exact a terrible price. And, as it has always been, it's up to us in the technology field to keep those two polar opposites in mind and design, engineer, and prepare accordingly.

"Alexa, set an alarm for five years." Let's talk about it then and see what's changed.

By the way, I'm doing more updates on Twitter and Facebook than ever before. Be sure to follow me on Twitter at @DavidGewirtz and on Facebook at Facebook.com/DavidGewirtz.

Newsletters

You have been successfully signed up. To sign up for more newsletters or to manage your account, visit the Newsletter Subscription Center.
See All
See All