Software testing lacking serious effort

In today's digital era, prior testing before software goes live is imperative, as its failure could damage a company's business and reputation. Considering fleeting brand loyalty today, negative user reactions can lead to user migration as well.

These potential consequences carry more significance than the technical malfunction, said observers who unanimously stressed the importance of software testing.

The approach of "test early, test often" is unfortunately not adhered to by most companies, and test efforts are usually left to the last minute, said Jeff Findlay, senior solution architect, Asia-Pacific and Japan at Micro Focus. Companies hence struggle to test critical functionalities and transactions before the software's release date.

Ray Wang, principal analyst and CEO of Constellation Research, added that even if companies acknowledge that getting software right is critical, they do so in "faster, better, cheaper" mode. This limits the possibility of software failure but also the foresight and preparation for unexpected glitches in real life.

"Testing has to be elevated into a science and not an art", Wang said, adding that only a handful of companies actually get this point of integrated, agile development. These companies build test plans side by side with functional specs, which takes more time to plan, but results in fewer bugs and errors, the analyst said.

"Just because you've counted all the trees doesn't mean you've seen the forest," said Rameshwar Vyas, CEO of Ranosys Technologies which offers software testing services. Companies should always bear this in mind while making a testing roadmap for any software release. Change management and risk management are also integral parts of the overall plan, so that all test cases regarding what the software is not supposed to do are covered as well, he advised.

Findlay emphasized that companies must ensure that what they test meets business requirements, which should be clearly defined, validated by all the stakeholders and kept in a central repository. Companies usually write disparate documents describing requirements differently and multiple times, resulting in a variety of interpretations which in turn leads to application failure and expensive rework, he explained.

These comments come after U.S. trading company Knight Capital Group made a trading loss of US$440 million after a botched software update made several erroneous orders on the New York Stock Exchange (NYSE) last Wednesday--and had to be bailed out with a financial lifeline. Barely a week later in Asia, the Tokyo Stock Exchange (TSE) experienced a computer failure in its backup systems which halted derivatives trading for 95 minutes on Tuesday.

How much is enough
How companies should gauge the amount and length of software testing to avoid such incidents boils down to how they associate the risk with a particular functions of the application, said Findlay.

It is important to understand the impact of failure on the business, and working backwards, mitigate those risks by ranking each test as early as possible so as to align these tests with the business requirement that drives them, the Micro Focus executive explained.

When transactions are critical to business success, they must be thoroughly tested from both a functional and nonfunctional perspective, he noted. This is to see if the application correctly performs the transaction in a timely fashion for all users, regardless of how they access the application.

Tests associated with non-critical aspects of an application don't require the same rigor and less testing is acceptable, since the business will not be seriously impacted in the event of failure, he added.

Even after the software goes live, continuous and regular testing of its critical transactions is important, Findlay noted. To facilitate this, automated tests should be developed as early as possible and re-run regularly to ensure development efforts don't break the application. These automated tests should cover the developer's code, functionality of key transactions and application performance, he advised.