Giving enterprise software practices an 'angioplasty'

With all the recent coverage of Web 2.0 and the enterprise, one of the things that isn't discussed enough is the way Web 2.0 is highlighting compelling alternatives to the usual ways of producing software.  One of the more recent write-ups in this vein was Stephen Bryant's Five Reasons Why Web 2.0 and Enterprises Don't Mix.  Stephen's points are generally right on and capture well the impedance that many larger organizations encounter when they try to take advantage of Web 2.0 ideas for themselves.

This gives some hope anyway that the innovation, rapid feedback, and low cost of current web development techniques will eventually translate to the enterprise.Tellingly, innate dislike of the "innovation from the bottom up" was one of the 5 items Stephen cites.  And I would argue that this is the point of departure for many organizations with Web 2.0 and other decentralized approaches like blogs, wikis, and even mashups.

This is the classic and unfortunate clash of culture that pull-oriented systems like Web 2.0 usually have when they encounter push-based systems like enterprise software development.  Interestingly, a recent report from the respected McKinsey and Company predicts that most innovation in the future will come from these types of pull systems, and that "as pull systems reach center stage, executives will have to reassess almost all aspects of the corporation."

The well-blogged about TIE event this month on Web 2.0 in the Enterprise also hit upon a bunch of these points.  One of the key observations being that Web 2.0 ideas encourage a fundamental change in business models much more than changes in technology. And that enterprises will have to learn how to give up some control to gain access to additional value.  It's along this seismic fault line where the bigger differences lie between the fast-moving, nimble world of Web 2.0 and the leviathan and often-ponderous processes of the enterprise.

The popular agile software processes like SCRUM and Lean Software Development have already attempted to address the bloat, excessive central control, and open loop nature of traditional software development. The old ways these methods seek to improve are too often surrounded with bureaucratic procedures that actively clog the productive channels of many organizations.  Or so goes the theory of agile development, which states that feedback loops and accountability from the real-world are vitally necessary todeliver software that does what is needed.

These days most folks have at least heard of the rapidly growing and popular agile software development movement.  The famed Agile Manifesto identifies the following points as being core to these ideals (and which we'll see are very similar to the Web 2.0 concepts):

Core Values of Agile Methods 

• Individuals and interactions over processes and tool
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

Now, let's look at the development ideals usually associated with creating Web 2.0 software:

Web 2.0 Development Ideals 

• Extremely Frequent Releases: Flickr sometimes does releases to production (to hundreds of thousands of users!) as often as every 30 minutes.  Benefits: Bugs go away faster, big bang integration stops, and the software experience becomes a continuous, smooth evolution.
  
• Small Pieces, Loosely Joined: Makes change easier, less risky.  The parts are also less specialized, more reusable, shareable, and hackable.
• Lightweight Programming Models:  Dynamic languages like Ruby and simple data formats like RSS, REST make development, integration, testing, and reuse easy and more cost-effective.
  
• Users as Co-Creators:  Involving and harnessing user participation is central to Web 2.0, whether that's just dynamically offering them the features they want, or allowing them to contribute information and attention to your online software.
• Real-time Feedback and Sampling: Learn about the features your users like by watching what they do and use the feedback to create an increasingly better product.
  

 As a further checkpoint, note that some Web 2.0 companies, particularly 37signals, have enumerated similar set of principles that are more closely aligned with lean software development, which is also based on the pull model:

• Less money: Encourages better use of resources and lower overhead and investment to recoup
• Less people: Less communication overhead, more focus on what you really need to produce, less cost
• Less time: Lots of time encourages wasted effort, loss of focus, more opportunities for disruption
• Less abstractions: Fewer Powerpoint charts, less unnecessary designs, less stuff "that isn't real"
• More constraints: Additional constraints like the ones above force more creativity and forces you to make more of what you have

The problem: How can this be applied to the enterprise?

The Web 2.0 world of software development is one comprised mainly of small startups and highly focused teams often driven by wild dreams of financial success.  I've heard it asserted, and personally experienced, that the hero-driven culture of startups and small companies just doesn't translate well into the corporate world of centrally planned capability and risk driven schedules.  And it certainly isn't repeatable.  Are the creation processes behind Web 2.0 software just not translatable?

The short answer is that we still have a chasm between two worlds.  On one hand we have small software creators using extremely dynamic methods, forced to be creative because of deliberately scant resources, and using rapid feedback loops to ensure their existing and potential customers get exactly what they want.  On the other hand, we have large, corporate software development and integration that is usually funded with significant budgets, detailed project plans, big design upfront (BDuF), and slow feedback loops to stakeholders.

Agile/lean methods, though increasingly popular in the enterprise, are often tame compared to Web 2.0 methods. 

I'm not going to attempt to answer the question of how Web 2.0 development techniques can be transferred to the enterprise here.  But certainly we've seen agile methods, which are closely related yet less extreme than Web 2.0 methods, scale to larger and larger projects in recent years.  And some agile experts, like Jutta Eckstein, now believe agile methods can be used effectively on projects with up to 200 people.

This gives some hope anyway that the innovation, rapid feedback, and low cost of current web development techniques will eventually translate to the enterprise.  But, if you read this excellent new report from the field by Marc Hedlund, you'll see that these new techniques might actually be accelerating away from existing practice.  So for now the jury is out, but you can be sure I'll be tracking this and discussing it here as we get more information from folks in the field.

Will Web 2.0's extreme agile approach bleed into the enterprise?  Do you have examples?