Questioning the reliability of an iPhone reliability study
(updated below)
There are some studies that just deserve the raising of a eyebrow, a cloud of doubt, a hint of skepticism. The latest study (PDF), conducted by a cell phone warranty company called SquareTrade, is the latest example of one that I can't help but question. In a nutshell, the company has concluded that iPhones are twice as reliable as Blackberries. (Techmeme)
One thing I learned in my many years as a journalist is that any study, survey, research report or poll can be skewed to reach any conclusion you'd like it to reach. And I can't help but wonder if there was some hidden agenda that prompted these findings to be stretched or tweaked to reach desired conclusions. I'm not saying that's what has happened here. But, when it comes to judging the reliability of a product, I tend to look at, well, how reliable it is. If there are hardware problems or software problems that are slowing down the system or causing calls to drop or Web pages to not load, then that device is not reliable. By definition, it cannot be relied upon to perform at a service level that was promised through marketing or sales channels.
Now, to be perfectly clear, I'm not saying that the Blackberry or the Treo are more reliable than the iPhone. I've had my share of problems with the Blackberry and, before that, a handful of different Treos. I'm also not saying that, a few years down the road, the iPhone won't be 2, 3, or even 10 times more reliable. But I have serious reservations about believing that the iPhone - after only a little over a year on the market - is TWICE as reliable. Give me a break.
But wait. There's more. According to the study, the call quality on the iPhone has fewer problems than the Blackberry and Treo. Now, this study is no longer making me pause; it's making me laugh.
In all fairness, I have no official data to counter SquareTrade's findings as it relates to the call quality on the iPhone. But judging from the word of mouth comments from friends and acquaintances around the country - as well as the dozens of comments from ZDNet readers who chimed in on an (unofficial) poll and blog post I put up last month - the quality of the AT&T voice service has been the biggest problem with the iPhone. That's just what I've heard. SquareTrade says that the biggest problem with the iPhone is touch screen issues. But the company also notes that "the iPhone is prone to failure due to accidental damage."
Duh. Aren't all electronic devices subject to failure due to accidental damage? Drop a PSP on a concrete floor and I'll be you have some problems with it. Drop any phone - even the cordless phones around the house - in a kitchen sink full of soapy water (I've seen that happen) and I'd be willing to guess that it might be subject to failure. Again, trying to be fair about it, the study found that "an iPhone is more than twice as likely to experience an iPhone failure due to damage than through a handset malfunction," largely because the device is slippery and could be dropped. (Hey, maybe this is a way to drum up sales of those non-slip covers for the iPhone.)
I know I'm coming down hard on SquareTrade and its findings but I have issues - not only as a journalist, but also as a consumer - with survey results that I feel insult my intelligence. In the study, there's a fair amount of forecasting about what might happen and a disclosure that the first generation iPhone had different issues, compared to the 3G models. I don't think that either of the models were out long enough for anyone - even a company that's in the business of selling extended warranties for such devices - to judge and draw conclusions.
In this case, I think word-of-mouth reviews from friends, co-workers and family members are still the best approach - at least while the product is still young.
update: I exchanged e-mails with Vince Tseng, VP of Marketing at SquareTrade, about this post. In a nutshell, he acknowledged that I made some valid points about the SquareTrade study but also took exception with a few of my statements. I understood his point of view and wanted to highlight some of his objections. They include:
- Nowhere in the report does it say "twice as reliable," though the study does note "that the reported iPhone malfunction rate is half that of the BlackBerry." The "twice as reliable" line was used in a several blog postings about the study.
- There was no hidden agenda, he said. "The purpose of this study is to a) help educate consumers about their phone options, and b) get the word out about SquareTrade," he said, adding that the study is actually bad for the iPhone segment of SquareTrade's business because it tells consumers that they don't really need SquareTrade's warranty.
- The study is about reported malfunctions, not about quality of network issues. The focus of the study was issues that render the phone unusable - things like non-functional microphones, inability to make outbound calls and so on.
- The results were for one year in and that the projected malfunction rates over two years anticipates that iPhone malfunction rates in the second year will be greater than the Blackberry.
In the end, Tseng and I agreed that the word "reliable" is a loaded word that maybe shouldn't have been used in the headline of the study. While I can appreciate Tseng's points about the tone of my post, I still maintain that a study that's judging the reliability or performance of a device like a cell phone needs to be complete - taking into account service and network issues, as well as hardware malfunctions. Otherwise, how else can consumers fully understand what they're up against before they make a purchase?