Certification: What's in a name?

The technology industry is awash with certifications at the individual and organisational level, but are these qualifications worth the paper they're printed on?

		Contents Weighing up the benefits of quality Certification costs, but who's buying? Hard to measure Case study Executive summary

The best thing about standards, the old saying goes, is that there are so many of them to choose from. The same could definitely be said about industry certifications, which in recent years have proliferated in number and scope to become major drivers for change within all sorts of businesses and service providers.
Or have they? Certainly, certification by third parties offers the promise that a company has been audited and found to meet certain required standards. Ditto individual certifications, which became hot commodities during the dot-com era, when talent was in high demand and often exorbitant salaries were frequently related to the number of letters after a candidate's name.

"Since the new edition of ISO 9000 was released in 2000, its come a long way -- but CMM is still the Bible to go to if you want to develop good internal processes."

Similarly, the past five years have seen the rise of company-wide certifications, such as ISO 9001 or the CMM (Capabilities Maturity Model) family of standards, as a way of differentiating a company's products and services to potential customers. Indian outsourcers have used this strategy to great effect, pursuing CMMI (CMM Integration) Level 5 certification with unparalleled vigour in order to create a perception that they stand for quality as well as low prices.
Before you let yourself become bedazzled by the array of framed certificates on the reception wall, however, it's important to remember that certifications may not always be what they've cracked up to be. Guidelines for assessment against known standards are widely available; some companies pursuing certification have been known to use these guidelines to fudge audits. Mountains of normally unused documentation may be produced specifically to pass an audit but sit fallow in everyday usage, while employees may be trained in how to pass an audit yet still lack the inherent quality philosophy that the standards try to imbue.

In other words, many companies may well be following the letter of the law, but not its spirit. Just ask Quentin Goldfinch, director of R&D with Avaya Labs Australia. In his years of experience, Goldfinch has worked with many companies where certification was approached more from a tick-the-box perspective than by actually building a quality culture. One person on his current team previously worked as a CMM assessor and saw many companies working hard specifically for the purpose of passing the required audits.

"Being manufacturing oriented, ISO was originally all about repeatability and it was process focused," he explains. "As long as your processes were within acceptable limits, you were in good shape. But it didn't particularly require companies to reach any particular quality standard, as long as you could produce mediocre quality continuously. Since the new edition of ISO 9000 was released in 2000, it's come a long way -- but CMM is still the Bible to go to if you want to develop good internal processes."

This disconnect is reflected in the bifurcated approach that Avaya Labs has taken towards quality certification. Part of global internetworking and voice systems vendor Avaya, the operation is certified to ISO 9001 quality standards -- now mandatory for supplying many government and commercial contracts -- but uses CMM's process and metric-driven principles to guide its long-term philosophical mindset.

"CMM improves our capabilities because project managers can decide what they need to do to get the best possible outcome from their projects."

It has not pursued formal CMM certification, however -- a point that Goldfinch explains away by arguing that an informal CMM approach allows the organisation to retain its flexibility.

"The fact we are using CMM because we want to means that we will use it the best way we can," he says. "It improves our capabilities because project managers can decide what they need to do to get the best possible outcome from their projects. We share that experience to other projects and across the organisation, and we have process groups that work across projects. We can pick and choose the best things out of CMM and -- instead of having 100 percent coverage of everything CMM -- can be more selective and value-driven in what we do."

Weighing up the benefits of quality
While CMM fever has typified India's outsourcing industry over the past decade, Australian companies have been far less enthusiastic about the certification. Here, just two firms have gone all the way to CMM Level 5 certification, with nominal numbers of CMM Level 4 companies and a very small number of CMM Level 3 and Level 2 certified organisations.

		Contents Certification: what's in a name? Weighing up the benefits of quality Certification costs, but who's buying? Hard to measure Case study Executive summary

Given its relatively low profile, there is no requirement yet for Australian companies to pursue CMM ideals to win corporate or government contracts. This could change in the long term, with some state governments said to be contemplating phasing in requirements that suppliers be quality certified.
For now, however, CMM remains a distant ideal for the majority of Australian companies. Perhaps it shouldn't be, however: a recent case series from Carnegie Mellon University's Software Engineering Institute, which developed and maintains the CMM body of standards, found that large adopters of CMM processes had delivered significant benefits.

At Boeing Australia, for example, a project to map development processes transferred CMMI Product Suite guidelines to the ISO 15504 (SPICE, or Software Process Improvement and Capability dEtermination) framework, an alternative method for assessing development processes. Over 18 months, Boeing reported a 33 percent decrease in the average cost to fix a defect, with time to turn around new releases cut in half and a 60 percent reduction in post-audit work.

Overseas, other providers were equally bullish about CMM. At General Motors, CMMI increased the percentage of milestones met during software development from around 50 percent to 95 percent; the average number of days late reduced from 50 to fewer than 10. US-based Lockheed Martin Management & Data Systems enjoyed a 20 percent reduction in unit software costs, and saw award fees -- given for standout contract performance -- increase by 55 percent between its achievement of CMM Level 2 in 1993 and full CMMI Level 5 certification in 2002.

IBM Global Services Australia has publicly said that integration of CMMI Level 5 ideals into its processes has saved more than $230 million for its customers over five years and increased development productivity by 76 percent. Of 1000 IBM projects undertaken during 2002, 98.5 percent were on time and 99.2 percent within budget; the number of production problems has decreased by nearly two-thirds, and the level of "high impact" problems by over 90 percent.

Other firms reported similar benefits. Accenture reported a 5:1 ROI for quality activities, while defence contractor Northrop Grummann Information Technology rated ROI at 13:1 (calculated as defects avoided per hour spent in training and defect prevention) after following CMM Level 5 processes during the construction of an inventory tracking system.

CMM also had an impact outside of the companies surveyed, with software development customers enjoying delivery of better code, faster. Aerospace firm Thales Training & Simulation, which achieved CMMI Level 2 compliance in 2001, noted that schedule variances decreased as its process maturity increased. Cousin organisation Thales Research & Technology, saw a 60 percent reduction in the cost of customer acceptance after progressing to maturity Level 3 of the Software CMM (SW-CMM), a predecessor to the current CMMI process.

These improvements don't come overnight, however: the Standard CMMI Appraisal Method for Process Improvement (SCAMPI) audit process is rigorous and intense. Accenture, for one, reported spending 8045 hours -- six months' solid work for a team of eight -- implementing the CMMI Product Suite. And each successive level of competence involves additional effort: the median time to move from CMM Level 1 to Level 2 is 24 months, according to Gartner. IBM's CMMI Level 5 certification required a review of 1100 developers in four cities, testing compliance with 425 distinct CMMI practices; its SCAMPI audit required five weeks' work from a team of nine auditors.

Clearly, compliance with CMM requirements translates directly into both cost savings (from shorter development times and fewer resources spent fixing problems) and smoother delivery for customers. It also offers intangible benefits, such as the creation of a quality development mindset that improves overall product delivery and innovation. Furthermore, having full CMM certification is still rare enough in Australia that it means something to potential customers.

While they may take CMM's ideals under advisement when planning quality activities, however, most Australian companies still appear to prefer ISO certification. Yet when every second company is ISO certified, presuming that the certification assumes a certain level of expertise, or ability to deliver, can be very problematic. Don't assume that ISO-certified suppliers are necessarily more effective than non-certified organisations; many smaller firms, particularly, lack the resources to commit to formal certifications but may well have developed a body of best practices upon which to call during their projects.

It is these practices, ultimately, that deliver the most value for customers of quality-certified organisations. Prudent assessment of tender bids, including a concerted effort to differentiate suppliers based on the experiences of their past customers, can be far more revealing than simply assuming that a particular certification assumes a particular level of competency. The key is to understand not how many certificates the potential supplier has on its wall -- but how many of the vendor's employees regularly read them.

Certification costs, but who's buying?
While its association with the growing Indian outsourcing sector has given it strong brand recognition, CMM is by no means the only certification that companies need to be thinking about. There are, for example, the above-mentioned SPICE, which is an ISO-standard process assessment methodology; TickIT, a UK/Sweden-derived software quality standard with global support; and Six Sigma, a manufacturing and service-focused quality process that has recently been extended to the world of software development.

		Contents Certification: what's in a name? Weighing up the benefits of quality Certification costs, but who's buying? Hard to measure Case study Executive summary

An outflow of the emphasis on software development process has been the promulgation of related project management best-practice standards. PRINCE (PRojects IN a Controlled Environment) and its successor PRINCE2 have become the UK's de facto project management methodologies, while many organisations have seen significant process improvements by following the Project Management Institute's PMBOK (Project Management Body of Knowledge), which addresses 40 key project management competencies.
There are also BS6079, a British Standard that guides general managers, project managers, project support staff, and educators; and the Japanese Body of Knowledge, which was recently released by the Japanese Project Management Forum and takes a more tower-like approach designed to facilitate management of multiple projects simultaneously.

With so many standards to choose from, it's not always clear what certification means -- either for companies choosing a methodology to follow, or for those assessing the qualifications of potential suppliers. And, as Avaya's experience highlights, many companies can absorb the benefits of certification without actually going through the bother and massive expense that's required for formal audits.

Indeed, financial limitations often mean that smaller companies are significantly disadvantaged when it comes to certification: the substantial auditing and documentation processes typical of formal quality standards require a level of detail and business process granularity that can often be missing in smaller companies. That's not to say that smaller companies can't benefit from measured process improvement, but rather that current large-scale quality religions have yet to accommodate the needs of smaller suppliers.

Software Engineering Australia, which has long taken a central role in encouraging Australian companies to pursue process improvement through standards such as CMMI, believes it has found a way to reconcile the growing need for demonstrated competency with the smaller resource base of SMEs, which make up the majority of software developers. That way is SEA's SoftwareMark, a certification program that combines many elements of CMMI with the European Software Institute's methodologies into a low-cost package for SMEs.

"We didn't feel like committing the resources involved to come up with a generic qualification to hang on our wall."

SoftwareMark was recently trialled successfully in Spain (where the ESI is based) and has attracted interest from national software bodies in Mexico, Bulgaria, and Taiwan. By bringing high-level project management competency to smaller companies, it could well become a global standard for software suppliers wanting to demonstrate competence without having to meet the almost Draconian requirements of heavyweight certifications like ISO and CMM.

"In the next two or three years, any company that's not adopting certification frameworks will get left behind," says Nathan Brumby, CEO of SEA. "It's more of a question of how it is represented and who pays for it. We are dealing with an industry that doesn't have enough money; for SMEs any money is too much money. That's one of the big inhibitors. And one of the challenges in Australia is that it's very hard to sell something here first -- but if you sell it somewhere else it seems to be a lot easier to sell here."

By focusing on both momentary and continuous process improvement, SoftwareMark could be particularly useful for SMEs, who lack the financial and people resources to make up for mistakes and inefficiencies during development. They also need formalised ways of reusing code and otherwise improving efficiency of the development process. This would make SoftwareMark the software equivalent of the Australian Wool Council's successful WoolMark certification program, providing an industry benchmark that becomes more meaningful as it's more broadly adopted. Furthermore, its lower cost makes it palatable for a much broader range of organisations.

Hard to measure
Given the wide-ranging interdependencies within Australia's software industry, encouraging smaller companies to gain formal certifications is in everybody's interest. Larger IT service providers could eventually come to prefer developers with SoftwareMark certification, knowing that as subcontractors those companies have the clear processes and internal strategies to get the job done quickly and effectively. Certainly, early suggestions are that SoftwareMark's guidance can be invaluable for small companies (see case study).

The same could be said for companies buying development services directly: certification implies efficiency that could translate into a higher chance of on-time and under-budget delivery, making a certified developer a lower-risk option than one that isn't certified.

It is, however, difficult to know just how much more to pay a certified provider as opposed to a non-certified provider. This problem has long plagued companies hiring individuals, for whom industry certification bodies have created so many formal certification programs that employers are no longer attaching such premiums to the skills.

Back in 2001, for example, a survey by US firm Foote Partners found that pay for holders of 53 industry certifications had increased 8 percent over the year earlier, while the premium pay for 82 more general technical skills was flat overall. The most recent Olivier Internet Job Index, however, noted a significant rise in demand for graduates (presumably, those less likely to have industry certifications).

"In the next two or three years, any company that's not adopting certification frameworks will get left behind."

The implications of this trend: many employers, disillusioned after hiring well-certified individuals that didn't necessarily deliver commensurate value, are choosing less experienced workers that they hope to train in specific methodologies suitable for their own businesses. Still others are looking for more general certifications in IT project management and quality control -- a distinct change from a few years ago when industry technical certifications were seen as guarantees that an individual could resolve complex technical and internetworking issues.

With businesses as with individuals, it is ultimately important not to assume that a supplier holding that certification will necessarily deliver a specific level of financial advantage. Many non-certified companies have invested heavily in developing repeatable and effective processes -- and may have developed methodologies that they see as providing a competitive advantage -- while others may have ticked all the right boxes but lack the philosophical change necessary to turn continual improvement into a constant motivator.

For now, evaluate potential suppliers based on whether they have repeatable processes in place, but focus more on the company's track record than whether or not they have a particular certification. Consider running a small pilot program to see how their processes are reflected in everyday work practice, then expanding the scope if they measure up. Companies that are walking the walk without talking the talk, so to speak, will quickly have their shortcomings revealed when they're put to the test.

Case study: SoftwareMark gets Geometry into better shape
As a small developer with experience supplying a largely government clientele with systems that rely on geographical spatial information, Hobart development house Geometry has found and exploited its market niche to good effect.

		Contents Certification: what's in a name? Weighing up the benefits of quality Certification costs, but who's buying? Hard to measure Case study Executive summary

Despite its history of success, however, the 10-person company's directors recognised that their ability to continue growing depended on building repeatable, effective development processes that focused on continuous improvement.
"We were keen to improve our own productivity levels, and we were also keen to expose ourselves to some level of international benchmarking to see how we did stack up, and where holes and opportunities for improvement were," explains Ashley Mahar, director of business development Geometry. "We had investigated things like ISO certification, but most were generic process certifications. We didn't feel like committing the resources involved to come up with a generic qualification to hang on our wall."

As in most SME development organisations, however, process change and eventual certification had proved to be far too expensive and complex an exercise given the company's natural financial and resource limitations. When the group learned about Software Engineering Australia's developing SoftwareMark program, however, it seized the opportunity to take a page out of industry best practice.

The effort was a major undertaking for the company, with the entire team mobilised to find and remedy process inefficiencies that had become endemic within many of Geometry's business activities. Over the course of more than eight months, the team gradually documented, audited, and improved its operations with the ever-present goal of providing more predictable, effective products for customers.

As a certification target, SoftwareMark proved to be remarkably broad, with coverage spreading from project management, quantification, and quoting of projects, breaking down work to ensure on-time delivery and avoidance of cost overruns. Many areas -- such as corporate governance and financial management -- are addressed even though they have little to do with the day-to-day development process. Across the board, business objectives are linked to project performance -- something that Mahar says was lacking in the past.

SoftwareMark certification has improved project performance tracking and business metrics, giving Geometry the confidence to pursue larger projects knowing that it has the processes in place to successfully cost and deliver the projects it promises. The resources Geometry has invested in SoftwareMark certification have, he continues, delivered very real benefits in terms of increased business and a better customer experience. And that, in the end, translates into long-term business growth.

"Small companies don't always have the breadth of customer base that exposes them to a lot of best practice," says Mahar. "You tend to stick to a small circle of customers. But I don't think there's any doubt that one could look at one substantial tender, at least, that we would have probably struggled to get had we continued in our old style. We've always been able to produce good systems, but we can produce them repetitively now. There's an atmosphere of calm and control in the company that wasn't as pervasive before."

Executive summary: looking past certifications

		Contents Certification: what's in a name? Weighing up the benefits of quality Certification costs, but who's buying? Hard to measure Case study Executive summary

That potential supplier or employee may seem to have the entire alphabet after their name, but that doesn't always guarantee they'll deliver top quality in your specific environment. Here are a few things to remember when playing the certifications game:
Technical certifications abound. It used to be that Microsoft, Sun, Novell, Cisco, and other leading IT vendors' certifications dominated the market, but independent bodies have complicated the game with a bevy of alternative certifications. Make sure you know what the certification actually represents, what areas of knowledge it involves, and whether it requires ongoing recertification to ensure skills stay current.
Quality is a culture, not a piece of paper. Certification to quality standards confirms that an organisation has passed audits, but it doesn't guarantee high quality in the long term, particularly for new types of engagements. That's only possible if the company's mindset has changed to embrace quality; all will say it has, but your own intuition will be invaluable in judging just how true this is.
Many of the best aren't certified. Most certification audits require a massive effort in improving, documenting, explaining, and demonstrating quality processes to audits. For many companies, particularly smaller ones, the sheer cost in manpower and resources means certification isn't worth it -- but they may still be an excellent choice of provider.
Proprietary certifications count, too. Many companies have developed their own development methodologies for addressing areas such as quality and consistency. Just because they don't bear an outside stamp of approval doesn't mean they don't work well; judge companies based on proven deliverables.
Everyone can learn from certification. Even if you're not ready to pursue formal certification, it can't hurt to have managers read through the documentation surrounding CMMI, Six Sigma, or other standards. The ideas contained therein may well stimulate innovative ideas within any company.
Staff development can backfire. Deciding how much to invest in staff certification is always difficult: underinvest, and they'll be underqualified and possibly disgruntled. Give them too many skills, and their market value increases. Given the potential benefits from upskilling, however, it's best to err on that side and focus on building HR and other policies that will make them want to stay.
Certification can't hurt your chances. Indeed, it could well become necessary as both private-sector and government organisations come to expect increasingly higher standards from suppliers. Demonstrate a commitment to quality, even if not to the letter of the standards, and you'll be able to pursue certification down the track should you have to.
Any certification can be good certification. Regardless of where it's from, most certifications confirm that a company or individual cares enough about the service they're offering to go the hard yards. That alone may be enough to justify hiring them, since they've shown they are willing and able to learn.
Benefit from their investment. Ultimately, certifications like CMM result in improved development efficiency, better project predictability, and a better return on your investment. Use this to your advantage: project predictability is invaluable, even if it means paying a bit more.
Smaller companies matter too. Most high-level certifications are aimed at the big end of town, but smaller companies may benefit even more from the direction and efficiencies they provide. Offerings like SEA's SoftwareMark package high-end discipline in an SME-friendly offering; in other environments, larger businesses may want to guide key SME suppliers through certification.

This article was first published in Technology & Business magazine.
Click here for subscription information.