If it's not poltergeists or the Russians, it could be your power utility
Four of the six battery backups that sacrificed themselves to save our gear.
Beep. "Honey."
[Sixty seconds go by.] Beep. "Honey."
[Another sixty seconds go by.] Beep. "Honey, something's beeping."
And thus began what was supposed to be the end of my evening. Here's one for your wall of weird. About 12:30am on Thursday, my wife and I finally finished up what had been a long work day. We and the pup had settled on the couch to watch some mindless TV before bed.
More great project ideas
Then my wife heard a beeping sound. Reluctantly, I got up off the couch and began triangulating. It turned out to be coming from my studio, the 9'x10' room where I do my webcasts.
The UPS that powered our 24-port Ethernet switch was complaining. It was a small APC battery backup UPS that was about four years old. It was late. I needed the network to be up and running the next morning for a stream of meetings.
To quickly solve the problem, I pulled another APC UPS out of service from another part of the room. That one wouldn't be needed for a few days, so I figured it would hold us over until I could order a new one.
So much for easy fixes. That second APC battery backup started beeping as soon as I plugged it into the outlet for the switch. It wouldn't work either. Thinking there was something wrong with the outlet, I pulled my trusty ten buck outlet tester out of my toolbox.
Even though the outlet tester told me the circuit was good, neither UPS worked.
By this time, my wife was fully involved in the problem. She recalled we have a "spare" UPS that protects the crock pot. We have power blips once in awhile, and on one sad occasion her Black Forest Pot Roast stopped cooking halfway through. Dinner was ruined. That UPS, another APC, is normally plugged into a GFI outlet just above the kitchen counter where it sits.
It wasn't beeping, and seemed to be working fine. We pulled it, brought it to the studio, and tried to get it to power the main house Ethernet switch, still hoping we could keep the switch running until we got a replacement UPS. Once we hooked it up in the studio, it started complaining, too. We wondered if the circuit for that room had gone bad.
We decided the next step would be to cycle the circuit breakers for the outlets that powered the failing UPS units. That didn't help.
Featured
About 20 minutes had passed. We were just starting to think we should troubleshoot by plugging the faulty UPSs into outlets on other circuits in the house when we heard more beeping, this time on the far side of the house. This time it was the APC battery backup in the hall closet that powers our home alarm.
So a fourth APC UPS had just stopped working and started beeping. None of them (except, perhaps, the crock pot unit) has ever had to handle much of a load. And yet, none of them work. Like the outlet in the studio, the outlet in the hall closet tested fine.
Now, before I go on, you need to understand that there had been no other indication of a power problem in the house that evening. Battery backup units from other brands, including the ones powering the main TV and my servers, were working just fine, as were a few from APC. But it seemed strange to us that all the units that were beeping were made by APC, and were all of a similar style and vintage (although they had different capacities).
We moved each of the four APCs around to different outlets in the house to test them. We knew these outlets were working fine because electronic items were plugged into them and operational. None of the UPSs worked, they all continued their beeping. It was starting to get to us.
By this time it was 2am. It was time to go to bed. I had early meetings in the morning. So we threw caution to the wind. We plugged the 24-port switch straight into the wall, hoping we would have a working network when we got up.
I had just completed my bedtime preparations and was walking towards the bedroom when -- suddenly -- two more APC battery backups started beeping. The one under our bed and the one near our couch lost their minds, too.
If you're keeping track, we're now up to six failed UPSs. All in one night. All made by APC. At this point, we were done. I had to get some sleep. We just pulled those UPSs from their outlets, shut them off, and went to bed.
After coffee this morning, I once again tried to figure out what could have happened. Remember, there had been no power failures. The other UPSs (mostly those that are NOT APC-brand) seemed to be working fine. All our power throughout the house also seemed to be working fine.
How was it possible that all six failed on the same quiet night? Why did it happen to only the APC battery backups and not any of the other-branded units? I'll be honest. By this point, my wife and I were beginning to suspect some engineered-in planned obsolescence design.
Far from it. As it turns out, those APC units died to protect us. But I didn't know it at the time. Like with all major, life threatening problems, I turned to the hive mind. I briefly described the problem on Twitter, Facebook, and to my fellow tech columnists of the Internet Press Guild.
One person suggested they'd been hacked by the Russians. Another suggested we might have poltergeists. But then, Stephen Satchell and Tom Henderson of the Internet Press Guild both suggested the problem might be the power coming into the building.
I had discounted that, because, as I said, we had no flickering or power loss. Also, none of the building power fault indicators on the UPS units were triggered or reporting errors. Even so, I decided to call Florida Power & Light (FPL), the public utility that provides our power.
The rest of the story
The representative at FPL was able to remotely query my meter. She noted that there was a telemetry report of a power spike at about 12:30am last night, just about the time the first of the UPSs started their beeping. She wasn't able to determine the cause remotely, so she initiated a truck roll.
A few hours later, the tech showed up at our house. This guy, Mike, has a job that would terrify me. Every day, he deals with voltage and current levels that, with a single mistake, could put him in the ground. And yet, that's what he does.
That's live power, right there.
When I'm sitting here in my sweat pants and T-shirt, whining because I don't feel like writing another briefing paper, article, or PowerPoint, I'm going to remember Mike. He puts his life on the line every day so we can watch our Netflix and zap our microwave popcorn. Just wow.
In any case, he pulled the meter off the outside wall and ran a series of tests. The lines to and from the transformer tested as solid and working. He did find a melted fitting on the roof, which he replaced.
He queried the logs for my meter, and confirmed that my 220 line took more than one hit of as much as 274 volts. That's about a 20 percent spike in voltage. That would certainly give my battery backups a headache.
Then, from his laptop, he started querying meters up and down my street. He found a number of the transformers had taken power surges last night. He also talked to another service tech, who reported a similar spike across town (but on the same substation) at about the same time. They concluded that something failed at the substation. That's what caused our ghosts in the machines.
What confused them initially was that they did not have any other customer calls in my neighborhood complaining about power problems. Then again, no one else probably has a house with 15 or 20 battery backups, never mind walls wired for Ethernet. Such is life as a full-time geek.
The moral of the story
The tech promised that FPL would immediately look into the substation failure. So far, so good. After he left, we ran out to the local warehouse club store with the best price and inventory, bought some new UPSs, and installed them. So far, no panicked beeps.
Here's a useful tip. Both the tech and the woman who took my call at FPL said I could file a claim to recover the cost of replacing those UPSs. While there are forms to file (and I'm not a fan of forms), I'm also not a fan of eating hundreds of dollars in UPS replacement costs. Kudos to both FPL employees for suggesting a way I might recover my costs.
Finally, let's talk about those APC UPSs. None were brand new. The prevailing theory (mine, the FPL tech, and the Facebook hive mind) is they were the UPSs closest to their mean time between failure, so they were the ones to fail.
Last night, my wife and I started out thinking the fault might be with the APC battery backup units. Today, we were humbled as it became apparent that they did their job. They gave up their lives to save us and our precious gadgets. They died as heroes.
I'm often asked whether it's worth having UPSs, especially since they can get costly. After last night, when I think about what might have been destroyed had those UPSs not bleeped out their hearts, I'm convinced the answer is yes. It's worth having them.
You can follow my day-to-day project updates on social media. Be sure to follow me on Twitter at @DavidGewirtz, on Facebook at Facebook.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, and on YouTube at YouTube.com/DavidGewirtzTV.