Tech Broiler

Jason Perlow and Scott Raymond

Dear Industry: Stop making us beta test your released products

By | December 22, 2011, 9:13am PST

Summary: Virtually every consumer electronics product I bought or used this year had significant issues that had to be fixed post-release.

As 2011 draws to a close, I wanted to bring up an issue that’s been bothering me all year, and that is that it seems that the entire notion of Quality Assurance has completely gone out the window with every major consumer electronics product that I’ve gotten my hands on.

I don’t want to pick on a single manufacturer per se, as this disease of “let’s throw it out there even though it isn’t quite ready yet” has become more or less an industry-wide epidemic.

Part of this problem I think can be attributed to the fact that our consumer products have become that much more sophisticated than in previous decades.

Popular products such as smartphones and tablets run their own embedded operating systems which pretty much make them full-blown computers in their own right, and each specific device needs to be QAed for device drivers and overall stability and performance before it is released to the general public.

In the case of products that use something like Android, you’re dealing with a generic OS or foundation that then needs to be modified or optimized to run on that product, which is comprised of different chipsets and component parts.

On top of that you have software stacks that include embedded applications that make the product experience unique or makes up the entire “value add”.

Because the market for consumer electronics has become so competitive, with so many players vying for attention in key growth areas such as tablets and smartphones, the pressure to get these products shipped is now greater than ever.

So what we have now is an systemic illness across the entire industry where we have these extremely sophisticated products that need a significant amount of QA with the added pressure of having to get them out the door fairly quickly.

This is a bad mix.

This is a seemingly incurable condition where manufacturers are now fully aware that their products may have to ship with known bugs or other significant problems in order to make deadlines to compete in the marketplace.

If there are significant issues, they fix them later with “over the air updates” for the poor bastards that took the leap and became early adopters.

The HP TouchPad is a prime example of a product that should have never been launched in the condition it was in. They actually sold this with excessive system logging turned on by default so the OS ran dog slow, requiring that you add 3rd-party patches to the thing in order to get acceptable performance.

The mainstream consumer electronics reviewers like David Pogue and Walt Mossberg at the New York Times and Wall Street Journal absolutely annihilated Hewlett-Packard in print for shipping such an abortion and the terrible reviews eventually resulted in the product being cancelled.

But had the TouchPad shipped with the performance boost that it received after its first update out of the gate, I think the reviews would have been a lot more positive, the sales might have been better and that product might not have been cancelled.

RIM’s PlayBook? Do I need to continue to harp on why this product should never have been released the way it was? No? Good.

But while the TouchPad and PlayBook are prime examples of botched QA at product launch, they weren’t the only ones. Virtually every Android tablet and smartphone that launched this year has suffered as a result of crappy QA by the OEMs and the carriers.

In the case of Android I have to lay most of the blame of that OS’s overall stability problems pretty much on Google. The OEMs and carriers only had so much to work with, and then they had to put their value add on top of an unstable foundation, which in this case was the first release of Honeycomb.

Pretty much every Honeycomb 3.0 tablet that was launched in 2011 was an unmitigated disaster in terms of overall stability, and they didn’t really get better until 3.1 and 3.2 came out later in the year.

This resulted in fairly high return rates for Android tablets, this despite the fact that from an overall hardware standpoint, they were pretty nice devices.

Virtually all of these tablets are still awaiting major updates, in the form of Android Ice Cream Sandwich 4.x.

Honeycomb may be the poster boy for the lousy reputation Android tablets had in 2011, but even products that used the “Stable” build of Android had issues. The Amazon Kindle Fire, which uses Android 2.3.x, was shipped with a bunch of known performance issues.

To give Amazon credit, these were addressed only a few weeks after launch with an update that was released this week. The update effectively has cleared up all of the known issues. But still, all of us who bought Kindle Fires were turned into unwilling beta testers.

Smartphones. Oh the smartphones. There’s far to many issues with these things to list in a single column, but I’ll hone in on the three major manufacturers that are really dropping the ball on quality control — Motorola, Samsung and Apple.

In the case of Motorola, I was one of the many unfortunate people that bought a Verizon Droid Bionic the week of its release. Now, overall I think the Bionic is a nice piece of hardware, and when it works properly, it’s a wonderful thing.

The problem is that at launch, it had a ton of of data communications issues, particularly with reception of LTE and 3G signal. If you check the Motorola support forums, you’ll see massive amounts of complaints about this phone.

Most of this can be attributed to the fact that the LTE code and firmware in Android devices is still beta quality. Verizon and Motorola has sent out an over the air update for the Bionic in the last week that addresses some, but not all of the connectivity issues.

Ice Cream Sandwich is supposed to be a catch-all for a lot of the issues that current generation Android phones are experiencing, although even on my Samsung Galaxy Nexus which uses Android 4.0, I’m having a lot of the same data connectivity issues.

And on the Nexus I also have to contend with random reboots and other quirkiness that you can expect from any .0 software release.

All of this is supposed to be fixed with yet again Android 4.0.3 — if you’re one of the lucky few that actually has a phone that is targeted for this software upgrade. Thanks for making me an unwilling beta tester, yet again.

And Apple? Okay, the company is doing a better job than most of the others, but the whole “batterygate” thing with the iPhone 4S could have been addressed during the product’s development cycle. As could have the various Wi-Fi and cellular reception-oriented issues which also were endemic to iOS-based products which received iOS 5 updates.

I have to wonder if Steve Jobs’ illness which led to his eventual passing this year caused Apple’s “ship it when it is ready” ethos to suffer, since he couldn’t always be on top of everything. Let’s hope that the company will recover quickly from its loss and instill better quality control in 2012.

Who else dropped the ball on quality control in 2011? Talk Back and Let Me Know.

Kick off your day with ZDNet's daily e-mail newsletter. It's the freshest tech news and opinion, served hot. Get it.

Topics

Jason Perlow, Sr. Technology Editor at ZDNet, is a technologist with over two decades of experience integrating large heterogeneous multi-vendor computing environments in Fortune 500 companies.

Disclosure

Jason Perlow

My Full-Time Employer is IBM. I write as a freelancer for ZDNet.

Disclaimer: The postings and opinions on this blog are my own and don't necessarily represent IBM's positions, strategies or opinions.

I own no investments or direct financial instruments in the companies I write about.

Biography

Jason Perlow

Jason Perlow, Sr. Technology Editor at ZDNet is a technologist with over two decades of experience with integrating large heterogeneous multi-vendor computing environments in Fortune 500 companies. A long-time computer enthusiast starting the age of 13 with his first Apple ][ personal computer, he began his freelance writing career starting at ZD Sm@rt Reseller in 1996 and has since authored numerous guest columns for ZDNet Enterprise and Ziff-Davis Internet. Jason was previously Senior Technology Editor for Linux Magazine, where he wrote about Open Source issues from 1999 to 2008.

In his spare time, Jason is an avid amateur chef and food writer, where his work reviewing New Jersey restaurants has appeared in The New York Times. He is also the founder of the popular food web site eGullet and blogs about restaurants and cooking at OffTheBroiler.com.

46
Comments

Join the conversation!

Just In

RE: Dear Industry: Stop making us beta test your released products
deaf_e_kate 26th Dec
@LBiege : or Microsoft...
In reality, Jason, isn't a first version release of ANY software or hardware/software product combo a beta product? Some just have less bugs than others.

What was that old axiom? Never install a "point zero" release of any operating system? How long has that bit of advice been valid?
0 Votes
+ -
My condolence.
@LBiege : or Microsoft...
Yes. That's it. Apple never had a bug or problem on Steve Jobs' watch. Absurd.
@dhmccoy I didn't say they never had a bug on his watch. Antennagate, obviously. But their overall quality control clearly declined in the last year.
0 Votes
+ -
BUt....
rhonin 22nd Dec
@jperlow
Who's vision was SIRI?
Steve or Tim?
It appears the Oct release was due to software issues not hardware.
So who approved the SIRI beta release?
@jperlow Ever since I can remember, first release Apple products have had problems, from Apple Mac IIs, through iMacs, MacBooks, Power Macs, you name it.

They are currently switching out old iPod Nanos due to a defect.

The number of graphic card bugs that the iMac and MacBook ranges have suffered after new hardware updates or new OS versions stopped being funny in 2008, but they still continue - Lion and certain AMD graphic chips being the last one I heard about.

Each version of the iPhone seems to have had technical problems.

And when we are talking pure QA, the iPhone has been a disaster here. We have 6 users and we have been through over 12 replacement phones in 9 months! Unfortunately, most of the replacements went to our CEO, who is no longer such an Apple fan, after having 4 iPhone 4s (plural, not 4S) replaced between May and July, with non-functioning antennes for BT and WLAN or dead batteries.

As you say in the article, Apple aren't the only culprit, but they have a long track record for botched releases, which are fixed after the fact.

On the other side, I've had 2 htc Mozart 7s this year, the first one was dodgy from the get-go and thankfully bricked itself with the first update. (Probably a dodgy processor or faulty memory, it wouldn't always accept calls, randomly crashed couldn't play the ring tones properly; the replacement worked flawlessly and accepted the update without any problems.)
  • Flagged
Siri. How could you mention companies that force their consumers to beta test their products without mentioning Siri.

iPhone 4S had 1 major feature that Apple markets more than any other feature: Siri. They don't advertise it as being a beta product, they push it as being the big differentiator between the iPhone 4S and every other smartphone. You have to navigate 3 pages deep before you get the tiny "beta" graphic.

So yes, Apple, stop making us beta test your released products.
@toddybottom, I think you missed the point. Perlow is not talking about stuff that is actually beta. I know that is a fine distinction for some but you probably understand the difference.
0 Votes
+ -
@YaBaby
While Apple does at some point, deep into the Apple website, with a tiny graphic, admit that Siri is a beta product (some would call it alpha) this does not excuse them from Perlow's rant. If Motorola put a comment on the 682,347th line of their code that the Xoom was a beta product, would that excuse them from our wrath?
@YaBaby Sorry, noone knew Siri as beta until it has crashed. Then they announced "but but but it was beta" oh yes , suuure...
0 Votes
+ -
That's a good point
William Farrel 22nd Dec
@AmediaN
noone knew Siri as beta until it has crashed. Then they announced "but but but it was beta"

That's appears to be Apple's new MO the past couple of years.

It's like they're living off their reputation, and don't feel the need to actually test before deploying their products anymore, as they know that quality is not important when you have people buying on hype.

Antenna, battery, proximity sensor, near bricking updates, it's just one thing after another..
Apple better get their act together as look what's happening to the last company that just lived off it's reputaion - Sony.

I guess I can now understand why Apple decide to become a patent troll.
  • Flagged
0 Votes
+ -
No defense of Apple here but...
use_what_works_4_U 23rd Dec
@YaBaby
I haven't used Siri and don't have an opinion on it. The fact of the matter is that Siri was very publicly called 'beta' at the iPhone 4s launch. Apple's website has always listed it as such VERY prominently http://www.apple.com/iphone/features/#siri

The ads on TV don't say it, and that is a shame, but Apple has done much more to point this out than Google's various products which are always released as 'public beta' and tend to remain that way. Go look at the graphic on the page I linked. If you can't see immediately that Siri is beta, then you're trying to ignore it.

If Siri has problems, then that's bad. But to say that Apple doesn't call it beta, or that they only do so "at some point, deep into the Apple website" is either willfully ignorant, or just plain old FUD baiting.

I'm not intending to defend Apple, but I am calling you out Toddy, for continuing on your blatantly biased trolling with false 'facts'. Grow up and make the slightest effort to confirm your data before you lie to our faces.
0 Votes
+ -
macadam: I stand by what I said
toddybottom 23rd Dec
"Apple's website has always listed it as such VERY prominently"

Really? Here? http://www.apple.com/ No. They've changed it from being "INTRODUCING SIRI" (no mention of beta) to now running a huge Siri ad (no mention of beta). After the commercial ends, it takes you back to the old home page. INTRODUCING SIRI. No mention of beta.

That's the home page and no mention of beta. So let's click on iPhone. It takes us here:
http://www.apple.com/iphone/
INTRODUCING SIRI. No mention of beta. That is the home page for iPhone. No mention of beta.

It isn't until you get 3 layers deep where they have a tiny little graphic that mentions Siri is beta. So macadam, you are calling me a liar yet what did I write?
"You have to navigate 3 pages deep before you get the tiny "beta" graphic".

So macadam, point out the lie. I stated as a fact that Apple hides the beta graphic 3 layers deep and that is exactly the truth. Where did I lie macadam?

You should have ended your post with this:
"No defense of Apple here"
@Toddybottom
Yes, your post was technically correct in that it takes 3 clicks to get to the product page IF YOU CHOOSE YOUR SPECIFIED PATH. On the other hand, if you google 'Siri' (which is how most internet users find their information after all) the very first link that is not a NASDAQ symbol takes you *directly* there. If you want to know about a product, and you are on the web you go to that product's page. Here, try it:
https://www.google.com/search?aq=f&sourceid=chrome&ie=UTF-8&q=Siri

I agreed with you that Apple could do better by putting the beta disclaimer in its ads, and I pointed out that it is a shame they don't. Where I take issue with you is the overall implication in your post that this information is being hidden or denied, neither of which is true. The vast majority of products in this article were released as finished, ready for primetime products and yet, according to the author anyway, they had beta type problems. You choose, as your example, the one major product to come out this year with very few flaws, that the majority of its users are happy with, and that was announced (and is labelled on its own product page) as beta, and try to sell us the premise that it is equivalent to these other situations. It's just not.

While technically you did not lie, your evidence is only true if everyone follows the exact steps that YOU outline. The implication that this information is being hidden is simply false. The route that most people take to find information on the Internet (Google search) takes you to the 'Beta' tag in one click. That's how I found it originally, and that's how my Mom, Dad, siblings, friends, and coworkers are most likely to find it.

Again, I would prefer that the Siri ads all said 'beta' but they don't. Apple is not, as you imply, hiding the fact. It's right on the product page, and it was very clearly stated in the product announcement. They are also not "making us" do anything. The iPhone 4S is a very capable smart phone without siri, and you don't have to buy it to begin with. You made a case that Apple is hiding information. Apple is very clearly not doing that. To my mind the overall message of your post (as with many of your posts) is a lie.
@toddybottom Siri was almost certainly omitted because Apple advertises it as being beta, releasing beta software to general consumers isn't Apple's habit, and (surprise!) Siri actually works as advertised. Without that latter characteristic, you might have a point, but sadly you don't. Yuck-yuck-yuck! Scrood up again! But thanks for the opportunity to point out how great Siri is, even in beta.
To the user, Siri may only appear to be "beta" because it's not available in all native languages in countries where the iPhone 4S is sold, nor does it support strong accents, nor does it absolutely always recognize what users wish it to do. More user input may be used to help train command recognition. Perhaps Apple is also training its own voice recognition technology to replace Nuance's.
0 Votes
+ -
How do you expect
Mcleary316 22nd Dec
@toddybottom apple to test something like Siri without without just pushing out the beta to have the mass public test it? Its the best way to test something like Siri and IMO it works pretty well.
@msalzberg
That orange box only appears on the Apple website. None of the TV commercials say a word about it. None of the advertising materials sent to the providers say a word about it. For the vast majority of the public, they only see the commercial, they don't go sit and read all the details on the Apple website.
@toddybottom have you actually spent any time using Siri? I have an iPhone 4s. Siri works just fine. I do not in any way feel I am a beta tester. I think the reason it is labelled beta is because by its very nature it will tend to improve rapidly as more and more people use it. But rest assured it is good even now.
It has also happened to me with 2 Nokia phones: N95 NAM and N97.

The N95 had a very buggy firmware on its release. At least they fixed most of the issues after "only" around 6 months with an update (v22?).

The N97 (not the N97 Mini) had a near unusable firmware. Up to 4 (or 5?) updates fixed a few issues (though not the stability problems) and added some. At the end, Nokia silently stopped releasing updates for it. Never officially admitted that the hardware was at fault (underpowered processor, too small RAM, too small C drive), that because of that it would never work correctly.

Companies should at least be responsible for any problems their beta-quality products give to consumers.
0 Votes
+ -
Good article but ...
Oknarf 22nd Dec
You might want to re-write the sentence below, probably got missed in QA since the article was rushed out and you rely on your readers to do the work that you or an editor could have / should have done prior to printing. Very much in keeping with the theme or your article don't you think? So it's not so much an industry problem as a cultural one. From Consumer Electronics to the media that review them. Funny coincidence.

"In the case of Android I have to lay the most of the blame of that OS???s overall stability pretty much on Google."
@Oknarf
I would agree 100% that it is a cultural shift. Managers want things done faster and faster and they also want to save money. In order to make stuff cheaper and build it faster you have to cut down on the quality. They won't tell you to cut corners but that's implicit in the demand for a product moving out the door faster and for cheaper. I've had this argument in meetings many times as people push to move work along faster and faster to meet even more unrealistic schedules. I challenge them and ask them, right to their face, if they want a low quality product. They say no. I ask them if they're giving me more staff ... they say no. I tell them I can't put out the same quality, faster, with the same people.

Apparently I'm uncooperative.
I don't really see this year as much different than any previous year. Computers have had issues since their inception and now that smartphones are easily as complicated as computers, if not more so, you're facing the same problems. Apple is managing to put out slightly more reliable products because they're doing everything in-house so compatibility is fairly simple. Yet, they're having issues because they can't predict all possible factors. One of the problems that Apple has, for instance, is their secrecy and the fact that they don't give the carriers very long with the devices before launch. The AT&T "inside look" at certifying new devices shows why Apple's method can potentially cause problems that Apple is unaware of. The various providers have methods to test it across the entire range of their network. Apple really doesn't have that sort of gear to work with.

That said, I find that the quality issues with Honeycomb were far overblown. I've used a Xoom since launch and had less problems with it than I have with my work laptop.
I do believe that Apple does the best pre-launch testing in the industry. It also appears to be true that Apple is held to a higher standard than others, especially Android-based devices. The "Antenna-gate" issue is a good example. This was seriously overblown because it was the iPhone. All cellphones suffer to some degree from signal loss when held in the human hand.

The only truly beta products that I know Apple to have launched are Apple TV, which they acknowledge is a "hobby" product, and Siri. Apple made it clear that Siri was a beta product from the beginning and necessarily so - it takes millions of unique users to establish the volume of data required to make artificial intelligence work as it should.

As for Android-based products, I don't see how it will ever get out of the beta launch cycle as long as Google can continue to make its billions from advertising regardless of the quality of the software. They have no monetary incentive to make Android as good as iOS (or WP7/8).
@thorntondw
The antennagate thing wasn't JUST because it was Apple, it was also because the results were worse than other phones and because of Apple's constant advertising of "it just works". Add to that the dismissive "you're holding it wrong" attitude ... and that's why it gained such traction.
My wife's Motorola Flipside (a.k.a. the "flip-off" or the "crapside") should NEVER have been released. It was a half-decent phone for the first few weeks, until the problems began. AT&T loaded so much crap that the memory left was well under 100 KB. Not MB. KB! It didn't take long before it started throwing "out of memory" messages. She got 4 or 5 apps to load before it just couldn't handle any more. The AT&T apps were NOT deletable without rooting the phone. Then the freezes and hangs started, along with instances of not being able to hear the party called or the called party not being able to hear her.

I just had to beg AT&T to let us sign a new 2 year agreement and give them $200 to get her a new phone. It was either that or live with this junk for another year. First time I've ever had a phone that didn't last as long as the contract....

This was my first Motorola in YEARS and it will surely be the LAST.
Isn't that whats referred to as being on the 'Bleeding Edge'... thats been around since the beginning of time.
As long as companies are competing to be the "first" to release a new product or service we'll be doomed to beta test release products.
the other day my work colleague told me: software companies are all Americans because they sell poor products that get improved. this model works. whereas German soft makers go bust after 5 yrs spent on the first version which never gets released as it is never perfect.
0 Votes
+ -
There's some truth to this
klumper 22nd Dec
@Drakkhen

Which begs the lament, you're damned if you do and damned if you don't.
0 Votes
+ -
Blaming 'the industry' is simple and satisfying. But the industry does not exist in an open space, it caters for us, the consumers. We need to take at least some of the blame for jumping for cheap, flashy, new toys.
Funny, my Windows Phone 7 has been flawless since I purchased it a year ago. I'm not used to Microsoft releasing such solid software.

Android is a mess, and always will be. It feels like Microsoft products 10 years ago. I will never invest in an Android device until Google cleans up its act.
@spaulagain Yeah, I guess an interface designed for retarded people has an appeal for some.
I blame consumers that just can't wait to own the latest XXXX. As with almost any code, give it six months in the wild before buying it, unless of course you are the guy that has to first on the block to have one.
0 Votes
+ -
No mention of Microsoft
root12 22nd Dec
How convenient that you didn't mention Microsoft. Microsoft continue to get a free pass. These are certainly Microsoft golden days in the blogosphere where it's ok for Microsoft to do it, but wrong for others.
0 Votes
+ -
Contributr
@root12 Microsoft did not release significant new consumer electronics products this year. No new XBOX, no new Kinect, etc. As to Windows Phone 7 Mango, it's essentially a bugfix release for the previous year's smartphones. There were only a few Mango-specific phones launched between all the major carriers.
Well said Jason,

in any other industry defective products would see class action suits abound however the software industry thinks it can get away with it using dodgy EULAs.
They want the full force of patent protection law for trivial and obvious code, fine then you are treated legally as any other consumer product, that might put paid to the silly patent wars at least.
Are you such a Apple fan boy that you mention it last? Antenna, battery, Siri , ...?
LOL! Most of the problems you mentioned are because of Google. Everything Google releases is a half-assed, half-baked beta product with tons of issues. It's why I constantly say Google is not a software company. They suck at software. Other than Google Search and maybe Google Chrome browser, their software has sucked nuts.
@jhammackHTH


+1

Google has taken the beta release to new heights and the worst part is the google fans tell everyone how great the google crap is. Does google even have a product that isn't "beta"?
Any Google product and Android, it is all beta.
The same thing happens with video games all the time. You might be right about the overall complexity being the reason, but IMO there isn't much reason besides the company just not wanting to spend the time and money when people would be buying and therefore "testing" the product anyway :/
0 Votes
+ -
The other side of the coin
perenic 23rd Dec
We can ask the industry to stop making us beta test of their released products. But we must then agree that we will wait longer for the polished products. Personally I rather live with a beta new product than no product at all (read: waiting for it to come out in two years). But that's me - an early adopter.
I've always been fascinated by the free pass that IT has received from the government and the judicial system in regard to its services and products. Everyone cries and complains about Big Oil, Big Pharm, Big Farm, and the like, but nary a peep when it comes to computer technology manufacturers. Why?
Is there any other industry that could get away with the abject shoddiness of its products without being sued out of existence? None that I know of. I work for the auto industry (in IT) and I can attest that if the windshield wipers of a vehicle are a quarter inch too large, then the feds will come down on the OEM like white on rice.
What's more, now that IT is firmly and wholly in the communcations business, how is it that they avoid the regulation that the phone and cable companies come under? Isn't the Internet now a 'vital' infrastructure component, such as insurance and utility companies?
The geeks have enjoyed the free ride on our dime too long. To them, 'innovate' means figuring out a way that I can't print a PDF on a Canon printer or inventing the latest way to hack my iTunes account. It's time they answer to Uncle Sam. Forced standardization will put a stop to this chaotic mess once and for all.

Join the conversation!

Formatting +
BB Codes - Note: HTML is not supported in forums
  • [b] Bold [/b]
  • [i] Italic [/i]
  • [u] Underline [/u]
  • [s] Strikethrough [/s]
  • [q] "Quote" [/q]
  • [ol][*] 1. Ordered List [/ol]
  • [ul][*] · Unordered List [/ul]
  • [pre] Preformat [/pre]
  • [quote] "Blockquote" [/quote]
ie8 fix

The best of ZDNet, delivered

ZDNet Newsletters

Get the best of ZDNet delivered straight to your inbox

Facebook Activity

White Papers, Webcasts, & Resources
ie8 fix