Mobile app testing imperative, but not foolproof

Mobile app testing imperative, but not foolproof

Summary: Developers need to avoid treating end-users as beta testers and conduct pre-launch app testing to avoid critical reviews due to poor usability once released to market, analysts urge.


Software developers should focus on testing their mobile apps before launching them and not rely on end-users to catch inherent bugs or treat them as beta testers after launch, say analysts. Efforts to test these apps should continue post-launch too, adds one developer.

Tim Shepherd, senior analyst at Canalys, said since mobile apps can go live relatively quickly compared to a full-fledged Web site, the practice of quick launches followed by constant updates to address any bugs reported back has enabled some sort of a crowd beta testing mindset leveraging end-users.

The fact that consumer feedback today is almost instantaneous only fuels this practice further, Shepherd added.

He said user feedback is always valuable, and developers that treat consumers as beta testers could experience a positive result in user acceptance testing (UAT) which helps in prioritizing the bugs to fix first. Conversely, it could backfire, and users get put off by the app's poor quality and usability, and decide not to download it, the analyst said.

For paid apps, in particular, consumers have a legitimate right to be unforgiving of poor quality and unwilling to re-engage with apps that disappoint, he pointed out.

"Developers need to be quick and responsive in addressing consumer concerns, but should do themselves a favor by preempting the most obvious complaints with appropriate pre-release beta testing," he urged.

Shepherd acknowledged that poor quality apps remain in the market because of the "tough economics" of app development. The market is rife with wildly differing revenues and costs but often offer slender profit margins, he explained.

That said, Aapo Markkanen, senior analyst at ABI Research, noted that the reasons for conducting pre-launch testing are as obvious as the drawbacks for not doing so. The more money involved in the project, the more important app testing becomes, he stressed.

"If there is a major bug that is not caught before the launch, then a lot of expensive development and marketing work is quickly wasted," Markkanen stated. "The first wave of users gives the app bad ratings, which then undermine [the number of] downloads and the app would [eventually] sink into oblivion."

Should the initial launch phase get botched, the app will need to be unique enough for consumers to give it another chance, he added.

Not foolproof method
Shepherd noted while pre-launch app testing is necessary, there is no foolproof framework to ensure a problem-free app once it is released publicly. "Some things will always be missed and no app is ever perfect," he said, adding that various developers adopt differing app testing methods anyway.

A basic guideline would involve "function, security, and polish", the Canalys analyst suggested. Elaborating, he said the app should function correctly and consistently and not put any user data at risk of leaks or unauthorized access. As for aesthetics, images and text should at the least be displayed correctly on different screen sizes.

Markkanen added that developers will have to find their own balance between conducting a thorough app test and other factors such as available resources, market competition, and the target audience for the product. For instance, the heterogeneity of Android devices and operating systems (OSes) means it is pointless to make an app fully compatible with every available device so developers need to decide which ones matter.

Reiterating Shepherd's point, the ABI Research analyst said there is no standard or optimal framework, in particular time, for testing apps. The complexity of the app would determine this aspect, with "simple, app-ified Web sites" requiring zero to little testing time whereas games might require more time.

Chua Zi Yong, CEO of app development startup Stream Media, said testing and debugging typically take up around 30 to 40 percent of an app's total development time. If the complexity of the app increases, it could go up to around 50 to 60 percent, he added.

Companies culpable of not paying as much attention by setting aside enough budget or time for testing are usually non-IT ones, he noted. Should the development be outsourced to a third-party vendor, then these constraints may result in unrealistic expectations projected on the vendor to deliver fully-functional apps.

The focus on testing should not end after the launch of an app either, Chua urged. "Companies cannot assume that the same app will work with every OS upgrade. Testing and updating are equally important even after launch," he said.

Topics: Apps, Mobility, Software Development

Jamie Yap

About Jamie Yap

Jamie writes about technology, business and the most obvious intersection of the two that is software. Other variegated topics include--in one form or other--cloud, Web 2.0, apps, data, analytics, mobile, services, and the three Es: enterprises, executives and entrepreneurs. In a previous life, she was a writer covering a different but equally serious business called show business.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • Testing is too expensive

    If you have an 99 cent or free app with only few hundred downloads then there is no reason to do much testing as app is pretty much dead. Without users or money for the app there is no reason to spend time testing or further developing it. My guess is that the number of apps that are not profitable is 95% or more. So there is no incentive to raise quality for 95% of apps.
  • Testing as profit incentive

    it's a chicken and egg thing,
    if you don't test, your application will likely to not work on many of the devices permutation out there.
    in that case, you will get bad review and indeed you will never be profitable.
    in many cases, not testing is even not an option if the application promote your service.
    most developers "tend" to believe Emulators represent the actual world and that's "Android is Android", but reality is that every OEM is trying is best to be more unique and release a "better" version of Android into the market. some with smaller or larger screen, some with other "uniqueness points"

    If you believe in what you are doing, make testing part of the development effort and cost. else you'll get bad reviews and don't make money (or even worth, risk your brand)

    Color:Black & White
    Family Line:Apple iPhone
    Camera:8.0 MP
    Carrier:Factory Unlocked
    Product Includes
    Apple iPhone 4S 64GB Unlocked
    Apple Earphones with Remote and Mic
    Dock Connector to USB Cable
    USB Power Adapter

    Motto:We believe in result and not reason's.
    Gift Nguyen
    • Ask for pics and price..

      pics can be sent to u..prices are wholesale price and negotiable on bulk purchase.
      Gift Nguyen
  • Wow

    as Barry said I didnt know that some one can make $7757 in 1 month on the internet. have you read this web page(Click on menu Home more information)
  • Testing is essential for the survival of a well built app

    PushToTest offers performance and stress testing for your app.

    • Link to PTT