Agile development, early warning, Darwin, and failure

Some IT projects are destined to fail due to haphazard planning, poor execution, or unexpected changes in the business or technical environment. Given the high incidence of failure, innovative methods for uncovering weak projects are always welcome. If discovered in time, risky projects can be fixed or killed; of course, either alternative is preferable to full-blown failure.

Agile development offers an approach to managing software projects that reduces risk of failure, among other benefits. Close cooperation between developers, users, and business experts is fundamental to the agile method, as described by the Agile Alliance, an industry group:

[C]lose collaboration between the programmer team and business experts; face-to-face communication (as more efficient than written documentation); frequent delivery of new deployable business value; tight, self-organizing teams; and ways to craft the code and the team such that the inevitable requirements churn was not a crisis.

Given the promise of faster development combined with lower cost and higher quality, it's no wonder agile has rapidly gained popularity. The following table summarizes research sponsored by the Agile Alliance, showing why developers are adopting this methodology:

The original Agile Manifesto conveys its philosophical underpinnings:

In an interview with ComputerWorld magazine, Kent Beck, inventor of Extreme Programming and a strong proponent of agile development, offered comments connecting agile and screening for risk:

[T]here's the problem in which you have to make sure your software works. You say that's something that you should be able to take for granted, but that has not been so. Why has that been the case?

Well, I think it's a combination of technical and social factors that leads to all the defects in deployed software.

There have been these studies that you're familiar with that talk about the fact that most software projects are failing. Is agile really a remedy for that, or is the jury still out?

Is agile a remedy for failing software projects? I think a lot of software projects should fail, and the problem is you just don't know which ones until you're pretty well into it [emphasis added]. So is agile a remedy for it? No. I think software projects are still going to fail because there still [will be] the promising ideas that don't work out in practice. One thing that agile development can give you is to make sure those projects fail faster, sooner, cheaper, so you can get on with the next thing.

Although Kent is referring to technology-driven software development efforts, where blind alleys are necessarily part of the process, let's push this issue in relation to business-oriented IT projects. If there's a Darwinian principle at work on IT projects, and many are doomed to fail, can we weed out the dogs sooner?

Most IT failures result from disconnects between technologists and business users, and the impact of failure is generally measured by negative effects on the organization, its finances, and users. Therefore, an early warning system for IT projects must rely on business-oriented criteria:

1. Project goal. Will the project likely produce a useful and worthwhile result?
2. Success capability. Has the project been constructed to succeed?

Point one appears straightforward, since no reasonable person would begin a purposeless project. To evaluate this assertion, let's define "worthwhile project goal" to mean "business case," since we're talking about business projects. Then, let's break business case down into tangible components:

• Functional scope. Has the project team clearly articulated what the project is supposed to accomplish.
• Business purpose. Will the project provide a tangible business benefit, or are the benefits hard to quantify?
• ROI. Have hard ROI numbers been calculated?

Unfortunately, many projects would not pass this test. Surprising as it may seem, "project goal" is a good early-stage filter to help determine whether or not a project may fail.

Point two, success capability, is intended to ferret out downstream weakness and vulnerability. If the following issues are not addressed early in the project, the risk of failure will be high:

• Stakeholder engagement. Users should be asked what they hope to gain from the proposed project.
• Executive sponsorship. In virtually all organizations, senior-level backing is required to make projects successful.
• Project management. Well-organized management is a critical success factor.
• Change management. When new software is introduced into an organization, the associated business processes often change. Users need assistance to help ease the transition.
• Resource availability. Especially on a large project, resources may be drawn from both internal business units and outside vendors, so this can be a tricky issue to manage.

Many failures would be avoided if these issues were honestly and objectively evaluated before starting the project. Quoting again from ComputerWorld, Kent Beck commented on honesty:

I think what's driving agile development right now is that it's possible to be much more honest, transparent and accountable if you have short cycles and you decide that that's what you want to accomplish. There's a huge latent market for software development that's just flat-out honest.

My corollary: honest, transparent, and accountable IT implementations rarely fail.

IT success depends on close cooperation between the technical and business sides of the project equation. The rules and practices of Extreme Programming imply a closer relationship between developers and consumers of software:

Here's what extremeprogramming.org says on the subject of programmers and users working together:

Customers enjoy being partners in the software process and developers actively contribute regardless of experience level.

Many failures remain undetected until they have run dramatically over-budget or schedule. Rather than relying on natural selection to weed out bad projects late in their lifecycle, it's far better to evaluate projects early, quickly stopping the ones that are destined to fail and fixing those that should be saved.

Although no methodology offers a silver bullet, systematic efforts can and should be made to understand the causes and conditions under which failures arise. Stopping failure before it occurs is the best way to succeed.

Update: For further discussion that places agile in context and perspective, be sure to view the video referenced in this post.