Dear Industry: Stop making us beta test your released products

Dear Industry: Stop making us beta test your released products

Summary: Virtually every consumer electronics product I bought or used this year had significant issues that had to be fixed post-release.


As 2011 draws to a close, I wanted to bring up an issue that's been bothering me all year, and that is that it seems that the entire notion of Quality Assurance has completely gone out the window with every major consumer electronics product that I've gotten my hands on.

I don't want to pick on a single manufacturer per se, as this disease of "let's throw it out there even though it isn't quite ready yet" has become more or less an industry-wide epidemic.

Part of this problem I think can be attributed to the fact that our consumer products have become that much more sophisticated than in previous decades.

Popular products such as smartphones and tablets run their own embedded operating systems which pretty much make them full-blown computers in their own right, and each specific device needs to be QAed for device drivers and overall stability and performance before it is released to the general public.

In the case of products that use something like Android, you're dealing with a generic OS or foundation that then needs to be modified or optimized to run on that product, which is comprised of different chipsets and component parts.

On top of that you have software stacks that include embedded applications that make the product experience unique or makes up the entire "value add".

Because the market for consumer electronics has become so competitive, with so many players vying for attention in key growth areas such as tablets and smartphones, the pressure to get these products shipped is now greater than ever.

So what we have now is an systemic illness across the entire industry where we have these extremely sophisticated products that need a significant amount of QA with the added pressure of having to get them out the door fairly quickly.

This is a bad mix.

This is a seemingly incurable condition where manufacturers are now fully aware that their products may have to ship with known bugs or other significant problems in order to make deadlines to compete in the marketplace.

If there are significant issues, they fix them later with "over the air updates" for the poor bastards that took the leap and became early adopters.

The HP TouchPad is a prime example of a product that should have never been launched in the condition it was in. They actually sold this with excessive system logging turned on by default so the OS ran dog slow, requiring that you add 3rd-party patches to the thing in order to get acceptable performance.

The mainstream consumer electronics reviewers like David Pogue and Walt Mossberg at the New York Times and Wall Street Journal absolutely annihilated Hewlett-Packard in print for shipping such an abortion and the terrible reviews eventually resulted in the product being cancelled.

But had the TouchPad shipped with the performance boost that it received after its first update out of the gate, I think the reviews would have been a lot more positive, the sales might have been better and that product might not have been cancelled.

RIM's PlayBook? Do I need to continue to harp on why this product should never have been released the way it was? No? Good.

But while the TouchPad and PlayBook are prime examples of botched QA at product launch, they weren't the only ones. Virtually every Android tablet and smartphone that launched this year has suffered as a result of crappy QA by the OEMs and the carriers.

In the case of Android I have to lay most of the blame of that OS's overall stability problems pretty much on Google. The OEMs and carriers only had so much to work with, and then they had to put their value add on top of an unstable foundation, which in this case was the first release of Honeycomb.

Pretty much every Honeycomb 3.0 tablet that was launched in 2011 was an unmitigated disaster in terms of overall stability, and they didn't really get better until 3.1 and 3.2 came out later in the year.

This resulted in fairly high return rates for Android tablets, this despite the fact that from an overall hardware standpoint, they were pretty nice devices.

Virtually all of these tablets are still awaiting major updates, in the form of Android Ice Cream Sandwich 4.x.

Honeycomb may be the poster boy for the lousy reputation Android tablets had in 2011, but even products that used the "Stable" build of Android had issues. The Amazon Kindle Fire, which uses Android 2.3.x, was shipped with a bunch of known performance issues.

To give Amazon credit, these were addressed only a few weeks after launch with an update that was released this week. The update effectively has cleared up all of the known issues. But still, all of us who bought Kindle Fires were turned into unwilling beta testers.

Smartphones. Oh the smartphones. There's far to many issues with these things to list in a single column, but I'll hone in on the three major manufacturers that are really dropping the ball on quality control -- Motorola, Samsung and Apple.

In the case of Motorola, I was one of the many unfortunate people that bought a Verizon Droid Bionic the week of its release. Now, overall I think the Bionic is a nice piece of hardware, and when it works properly, it's a wonderful thing.

The problem is that at launch, it had a ton of of data communications issues, particularly with reception of LTE and 3G signal. If you check the Motorola support forums, you'll see massive amounts of complaints about this phone.

Most of this can be attributed to the fact that the LTE code and firmware in Android devices is still beta quality. Verizon and Motorola has sent out an over the air update for the Bionic in the last week that addresses some, but not all of the connectivity issues.

Ice Cream Sandwich is supposed to be a catch-all for a lot of the issues that current generation Android phones are experiencing, although even on my Samsung Galaxy Nexus which uses Android 4.0, I'm having a lot of the same data connectivity issues.

And on the Nexus I also have to contend with random reboots and other quirkiness that you can expect from any .0 software release.

All of this is supposed to be fixed with yet again Android 4.0.3 -- if you're one of the lucky few that actually has a phone that is targeted for this software upgrade. Thanks for making me an unwilling beta tester, yet again.

And Apple? Okay, the company is doing a better job than most of the others, but the whole "batterygate" thing with the iPhone 4S could have been addressed during the product's development cycle. As could have the various Wi-Fi and cellular reception-oriented issues which also were endemic to iOS-based products which received iOS 5 updates.

I have to wonder if Steve Jobs' illness which led to his eventual passing this year caused Apple's "ship it when it is ready" ethos to suffer, since he couldn't always be on top of everything. Let's hope that the company will recover quickly from its loss and instill better quality control in 2012.

Who else dropped the ball on quality control in 2011? Talk Back and Let Me Know.

Topic: CXO


Jason Perlow, Sr. Technology Editor at ZDNet, is a technologist with over two decades of experience integrating large heterogeneous multi-vendor computing environments in Fortune 500 companies. Jason is currently a Partner Technology Strategist with Microsoft Corp. His expressed views do not necessarily represent those of his employer.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • RE: Dear Industry: Stop making us beta test your released products

    In reality, Jason, isn't a first version release of ANY software or hardware/software product combo a beta product? Some just have less bugs than others.

    What was that old axiom? Never install a "point zero" release of any operating system? How long has that bit of advice been valid?
    • Poor guy must be using Google stuff

      My condolence.
      • RE: Dear Industry: Stop making us beta test your released products

        @LBiege : or Microsoft...
  • RE: Dear Industry: Stop making us beta test your released products

    Yes. That's it. Apple never had a bug or problem on Steve Jobs' watch. Absurd.
    • RE: Dear Industry: Stop making us beta test your released products

      @dhmccoy I didn't say they never had a bug on his watch. Antennagate, obviously. But their overall quality control clearly declined in the last year.
      • BUt....

        Who's vision was SIRI?
        Steve or Tim?
        It appears the Oct release was due to software issues not hardware.
        So who approved the SIRI beta release?
      • RE: Dear Industry: Stop making us beta test your released products

        @jperlow Ever since I can remember, first release Apple products have had problems, from Apple Mac IIs, through iMacs, MacBooks, Power Macs, you name it.

        They are currently switching out old iPod Nanos due to a defect.

        The number of graphic card bugs that the iMac and MacBook ranges have suffered after new hardware updates or new OS versions stopped being funny in 2008, but they still continue - Lion and certain AMD graphic chips being the last one I heard about.

        Each version of the iPhone seems to have had technical problems.

        And when we are talking pure QA, the iPhone has been a disaster here. We have 6 users and we have been through over 12 replacement phones in 9 months! Unfortunately, most of the replacements went to our CEO, who is no longer such an Apple fan, after having 4 iPhone 4s (plural, not 4S) replaced between May and July, with non-functioning antennes for BT and WLAN or dead batteries.

        As you say in the article, Apple aren't the only culprit, but they have a long track record for botched releases, which are fixed after the fact.

        On the other side, I've had 2 htc Mozart 7s this year, the first one was dodgy from the get-go and thankfully bricked itself with the first update. (Probably a dodgy processor or faulty memory, it wouldn't always accept calls, randomly crashed couldn't play the ring tones properly; the replacement worked flawlessly and accepted the update without any problems.)
  • You missed Apple's biggest failing in this department

    Siri. How could you mention companies that force their consumers to beta test their products without mentioning Siri.

    iPhone 4S had 1 major feature that Apple markets more than any other feature: Siri. They don't advertise it as being a beta product, they push it as being the big differentiator between the iPhone 4S and every other smartphone. You have to navigate 3 pages deep before you get the tiny "beta" graphic.

    So yes, Apple, stop making us beta test your released products.
    • RE: Dear Industry: Stop making us beta test your released products

      @toddybottom, I think you missed the point. Perlow is not talking about stuff that is actually beta. I know that is a fine distinction for some but you probably understand the difference.
      • Siri is not marketed as a beta product

        While Apple does at some point, deep into the Apple website, with a tiny graphic, admit that Siri is a beta product (some would call it alpha) this does not excuse them from Perlow's rant. If Motorola put a comment on the 682,347th line of their code that the Xoom was a beta product, would that excuse them from our wrath?
      • RE: Dear Industry: Stop making us beta test your released products

        @YaBaby Sorry, noone knew Siri as beta until it has crashed. Then they announced "but but but it was beta" oh yes , suuure...
      • That's a good point

        [i]noone knew Siri as beta until it has crashed. Then they announced "but but but it was beta"[/i]

        That's appears to be Apple's new MO the past couple of years.

        It's like they're living off their reputation, and don't feel the need to actually test before deploying their products anymore, as they know that quality is not important when you have people buying on hype.

        Antenna, battery, proximity sensor, near bricking updates, it's just one thing after another..
        Apple better get their act together as look what's happening to the last company that just lived off it's reputaion - Sony.

        I guess I can now understand why Apple decide to become a patent troll.
        William Farrel
      • No defense of Apple here but...

        I haven't used Siri and don't have an opinion on it. The fact of the matter is that Siri was very publicly called 'beta' at the iPhone 4s launch. Apple's website has always listed it as such VERY prominently

        The ads on TV don't say it, and that is a shame, but Apple has done much more to point this out than Google's various products which are always released as 'public beta' and tend to remain that way. Go look at the graphic on the page I linked. If you can't see immediately that Siri is beta, then you're trying to ignore it.

        If Siri has problems, then that's bad. But to say that Apple doesn't call it beta, or that they only do so "at some point, deep into the Apple website" is either willfully ignorant, or just plain old FUD baiting.

        I'm not intending to defend Apple, but I am calling you out Toddy, for continuing on your blatantly biased trolling with false 'facts'. Grow up and make the slightest effort to confirm your data before you lie to our faces.
      • macadam: I stand by what I said

        "Apple's website has always listed it as such VERY prominently"

        Really? Here? No. They've changed it from being "INTRODUCING SIRI" (no mention of beta) to now running a huge Siri ad (no mention of beta). After the commercial ends, it takes you back to the old home page. INTRODUCING SIRI. No mention of beta.

        That's the home page and no mention of beta. So let's click on iPhone. It takes us here:
        INTRODUCING SIRI. No mention of beta. That is the home page for iPhone. No mention of beta.

        It isn't until you get 3 layers deep where they have a tiny little graphic that mentions Siri is beta. So macadam, you are calling me a liar yet what did I write?
        "You have to navigate 3 pages deep before you get the tiny "beta" graphic".

        So macadam, point out the lie. I stated as a fact that Apple hides the beta graphic 3 layers deep and that is exactly the truth. Where did I lie macadam?

        You should have ended your post with this:
        "No defense of Apple here"
      • RE: Dear Industry: Stop making us beta test your released products

        Yes, your post was technically correct in that it takes 3 clicks to get to the product page IF YOU CHOOSE YOUR SPECIFIED PATH. On the other hand, if you google 'Siri' (which is how most internet users find their information after all) the very first link that is not a NASDAQ symbol takes you *directly* there. If you want to know about a product, and you are on the web you go to that product's page. Here, try it:

        I agreed with you that Apple could do better by putting the beta disclaimer in its ads, and I pointed out that it is a shame they don't. Where I take issue with you is the overall implication in your post that this information is being hidden or denied, neither of which is true. The vast majority of products in this article were released as finished, ready for primetime products and yet, according to the author anyway, they had beta type problems. You choose, as your example, the one major product to come out this year with very few flaws, that the majority of its users are happy with, and that was announced (and is labelled on its own product page) as beta, and try to sell us the premise that it is equivalent to these other situations. It's just not.

        While technically you did not lie, your evidence is only true if everyone follows the exact steps that YOU outline. The implication that this information is being hidden is simply false. The route that most people take to find information on the Internet (Google search) takes you to the 'Beta' tag in one click. That's how I found it originally, and that's how my Mom, Dad, siblings, friends, and coworkers are most likely to find it.

        Again, I would prefer that the Siri ads all said 'beta' but they don't. Apple is not, as you imply, hiding the fact. It's right on the product page, and it was very clearly stated in the product announcement. They are also not "making us" do anything. The iPhone 4S is a very capable smart phone without siri, and you don't have to buy it to begin with. You made a case that Apple is hiding information. Apple is very clearly not doing that. To my mind the overall message of your post (as with many of your posts) is a lie.
    • RE: Dear Industry: Stop making us beta test your released products

      @toddybottom Siri was almost certainly omitted because Apple advertises it as being beta, releasing beta software to general consumers isn't Apple's habit, and (surprise!) Siri actually works as advertised. Without that latter characteristic, you might have a point, but sadly you don't. Yuck-yuck-yuck! Scrood up again! But thanks for the opportunity to point out how great Siri is, even in beta.<br>To the user, Siri may only appear to be "beta" because it's not available in all native languages in countries where the iPhone 4S is sold, nor does it support strong accents, nor does it absolutely always recognize what users wish it to do. More user input may be used to help train command recognition. Perhaps Apple is also training its own voice recognition technology to replace Nuance's.
    • How do you expect

      @toddybottom apple to test something like Siri without without just pushing out the beta to have the mass public test it? Its the best way to test something like Siri and IMO it works pretty well.
    • RE: Dear Industry: Stop making us beta test your released products


      What's that orange box say?
      • RE: Dear Industry: Stop making us beta test your released products

        That orange box only appears on the Apple website. None of the TV commercials say a word about it. None of the advertising materials sent to the providers say a word about it. For the vast majority of the public, they only see the commercial, they don't go sit and read all the details on the Apple website.
    • RE: Dear Industry: Stop making us beta test your released products

      @toddybottom have you actually spent any time using Siri? I have an iPhone 4s. Siri works just fine. I do not in any way feel I am a beta tester. I think the reason it is labelled beta is because by its very nature it will tend to improve rapidly as more and more people use it. But rest assured it is good even now.