Reuse, redundancy and the chasm 2.0

Reuse has been a constant and yet elusive goal throughout the history of software. Today it's a key part of the argument for adopting SOA, SaaS and Web 2.0. But I have a feeling that people approach it with one of two contrasting mindsets, only one of which is destined to bear fruit. I'm going to call them the creationist and the darwinist mindsets. I think you can probably guess which one I'm going to favor.

The creationist mindset abhors randomness and uncertainty.The key to crossing chasm 2.0 is a darwinist mindset When it sets out to build a software program, it aims to cover every use case and contingency. Its notion of reuse is to imagine a single, finite project to bring into being a core foundation set of software components or services, which from then on can be assembled on demand to meet virtually every conceivable need. This mindset is destined to be disappointed by the reality of reuse, when services designed with a view to sharing are frequently either ignored and never reused, or have their functions duplicated by subtly changed alternatives.

Of course the simple flaw when bringing a creationist mindset to the practice of software programming is that no developer or project team is blessed with the requisite supranatural omniscience to foresee every possible permutation and outcome. Yet recent research shows that a growing number of enterprise SOA projects are nevertheless built with a creationist mindset — or worse, cost-justified based on the imagined reuse — and are now experiencing disappointment or even censure at the poor levels of reuse achieved.

The darwinist mindset doesn't give a damn. Services come into being for their own autonomous reasons — to meet a specific project need, to win recognition for the creators, or in hopes of widespread adoption. It doesn't matter why, since there is no overarching masterplan, just an ecosystem with some inherent boundaries that surround implementation and performance.

The darwinist approach is hugely wasteful, since many services never get used beyond the purpose for which they were originally designed, and some never even get used for that. The mindset is most prevalent in the Web 2.0 environment, where the vast majority of projects are speculative. It is frowned upon in the enterprise, although it should be noted that the creationist approach is no more successful in achieving reuse than a darwinist mindset, despite being funded more generously on the assumption that it will be. Recognizing the impossibility of guaranteeing reuse, the darwinist approach prizes economy of outlay.

One of the most interesting examples of a darwinist ecosystem in the SaaS world is Salesforce.com's Apex platform and AppExchange marketplace. At the launch of the Apex programming language last year, the company's EVP of technology Parker Harris threw out a firmly darwinist message to the assembled Dreamforce masses: "When we give this to you, what are you going to do with it?" Meanwhile, dissatisfied murmurings from a number of AppExchange partners demonstrates the harsh darwinist realities of that marketplace. There's no guarantee that any service or application introduced into the ecosystem will achieve any reuse. Only the fittest, in a strictly darwinian sense (ie most fit for purpose), will survive.

Salesforce.com can afford to ignore the mutterings, because redundancy and disuse is good for the overall health of the AppExchange ecosystem. The more options there are to choose from, the more likely it is that a few of them will turn out to be the winners that rise to the top of the pile. AppExchange itself has to survive in the wider ecosystem of the on-demand business Web, and can only do so if it successfully evolves the best set of services.

Can SAP put its NetWeaver SOA platform at the center of an equally darwinist ecosystem? I'm sure SAP and its customers and partners are prey to some very strong creationist tendencies, but there's plenty of evidence that the xApps ecosystem has a darwinist bent. Shai Agassi, president of SAP's technology and product group, has been talking about a target figure of 500 xApps available from partners, and there's been talk of partners complaining about poor customer traction for some xApps. The worry however is that it would simply not be viable for SAP to build enough redundancy into its platform to allow darwinist creativity to flourish — that the core business process services it is building will prove too inflexible.

A theme from Agassi's talk last December that Dan Farber highlighted was his ambition of SAP crossing the chasm by the end of 2007:

He believes that SAP is in position to lead the chasm-crossing from the client-server model to enterprise SOA, and become the dominant software platform. He cited Geoffrey Moore's Crossing the Chasm, in which the author states that the first company to cross the chasm — to transition from one technology generation to the next — can galvanize 80 to 90 percent of the ecosystem. An ecosystem doesn't want to adopt multiple platform standards, and would rather have one strong platform, one backbone running core processes, to build around.

While Crossing the Chasm is a great manual for success as a 1990s software company, I do wonder which elements of his chasm theory still apply to the Web 2.0 world. The book describes how, back in the late 1980s, software development was such an expensive business that startups wanting to find success in the enterprise software market needed to find a 'visionary' manager at an establisehd large enterprise who could command a big enough budget to fulfil his or her vision (the classic example was Sheldon Laube's adoption of Notes at Price Waterhouse).

I don't think that still applies today. Software development is cheap, especially when Web 2.0 developers can tap third-party services as ready-made components of their application. What's more, visionaries can test out their ideas with a fraction of the investment required back then, which means the pool of visionaries a technology entrepreneur can work with is much larger — and (this is crucial) can have a great deal more redundancy built into it. Look specifically at the equivalents of Notes in the Web 2.0 age — a host of startups as well as more established companies, all being tested out by visionaries scattered across all manner of enterprises, from large multinationals to small businesses to multi-participant virtual enterprises.

The chasm that Geoffrey Moore wrote about still exists, of course, as a gulf between early adopters and mainstream adoption that emerging technologies still have to cross. But in the Web 2.0 world, the strategy for developing the momentum to cross the chasm is markedly different. Darwinism emerges as a force much earlier in the process, so that instead of working with one or two specially chosen and privileged visionary creators to hone a perfectly-defined product, today's innovators harness the massive reach of the Web to work with crowds of visionaries. They don't care that many of those visionary ideas for their new product won't fly — it's a darwinist system, in which the more redundancy there is, the more likely it becomes that one or two successful visions will emerge and prove durable.

The crucial difference with chasm 2.0 is the democratization of software innovation in the Web 2.0 era. It's no longer the case that there are only a few enterprises with the scale and resources to sustain visionary innovations. The entire Web 2.0 ecosystem itself provides that scale and makes the resources easily affordable. Therefore the key to crossing chasm 2.0 is a darwinist mindset that can harness those crowds of visionaries and build sufficient momentum among them to then emerge into the mainstream.

Seen in that context, I feel that SAP's NetWeaver ecosystem looks creationist at heart, with just a few darwinist notions around the edges. Therefore I would argue that Shai Agassi's hopes of winning the ecoystem wars by getting across the chasm first with NetWeaver are misplaced. There is, certainly, a shift happening at the intersection of SOA, SaaS and Web 2.0 to a new technology generation. But I don't believe we can yet identify the platforms that will successfully cross chasm 2.0 to become the dominant players of that next generation.