Mobile app testing imperative, but not foolproof

Software developers should focus on testing their mobile apps before launching them and not rely on end-users to catch inherent bugs or treat them as beta testers after launch, say analysts. Efforts to test these apps should continue post-launch too, adds one developer.

Tim Shepherd, senior analyst at Canalys, said since mobile apps can go live relatively quickly compared to a full-fledged Web site, the practice of quick launches followed by constant updates to address any bugs reported back has enabled some sort of a crowd beta testing mindset leveraging end-users.

The fact that consumer feedback today is almost instantaneous only fuels this practice further, Shepherd added.

He said user feedback is always valuable, and developers that treat consumers as beta testers could experience a positive result in user acceptance testing (UAT) which helps in prioritizing the bugs to fix first. Conversely, it could backfire, and users get put off by the app's poor quality and usability, and decide not to download it, the analyst said.

For paid apps, in particular, consumers have a legitimate right to be unforgiving of poor quality and unwilling to re-engage with apps that disappoint, he pointed out.

"Developers need to be quick and responsive in addressing consumer concerns, but should do themselves a favor by preempting the most obvious complaints with appropriate pre-release beta testing," he urged.

Shepherd acknowledged that poor quality apps remain in the market because of the "tough economics" of app development. The market is rife with wildly differing revenues and costs but often offer slender profit margins, he explained.

That said, Aapo Markkanen, senior analyst at ABI Research, noted that the reasons for conducting pre-launch testing are as obvious as the drawbacks for not doing so. The more money involved in the project, the more important app testing becomes, he stressed.

"If there is a major bug that is not caught before the launch, then a lot of expensive development and marketing work is quickly wasted," Markkanen stated. "The first wave of users gives the app bad ratings, which then undermine [the number of] downloads and the app would [eventually] sink into oblivion."

Should the initial launch phase get botched, the app will need to be unique enough for consumers to give it another chance, he added.

Not foolproof method
Shepherd noted while pre-launch app testing is necessary, there is no foolproof framework to ensure a problem-free app once it is released publicly. "Some things will always be missed and no app is ever perfect," he said, adding that various developers adopt differing app testing methods anyway.

A basic guideline would involve "function, security, and polish", the Canalys analyst suggested. Elaborating, he said the app should function correctly and consistently and not put any user data at risk of leaks or unauthorized access. As for aesthetics, images and text should at the least be displayed correctly on different screen sizes.

Markkanen added that developers will have to find their own balance between conducting a thorough app test and other factors such as available resources, market competition, and the target audience for the product. For instance, the heterogeneity of Android devices and operating systems (OSes) means it is pointless to make an app fully compatible with every available device so developers need to decide which ones matter.

Reiterating Shepherd's point, the ABI Research analyst said there is no standard or optimal framework, in particular time, for testing apps. The complexity of the app would determine this aspect, with "simple, app-ified Web sites" requiring zero to little testing time whereas games might require more time.

Chua Zi Yong, CEO of app development startup Stream Media, said testing and debugging typically take up around 30 to 40 percent of an app's total development time. If the complexity of the app increases, it could go up to around 50 to 60 percent, he added.

Companies culpable of not paying as much attention by setting aside enough budget or time for testing are usually non-IT ones, he noted. Should the development be outsourced to a third-party vendor, then these constraints may result in unrealistic expectations projected on the vendor to deliver fully-functional apps.

The focus on testing should not end after the launch of an app either, Chua urged. "Companies cannot assume that the same app will work with every OS upgrade. Testing and updating are equally important even after launch," he said.