The growing primacy of data over software

I've been studying the transcript of this week's public conversation between Bill Gates and Tim O'Reilly at MIX 06.  I was at the conversation and yet even at the time I couldn't help thinking that O'Reilly and Gates were talking past each other on several key points. In particular, the increasing value of data and its potential as the primary source of lock-in and disruption in the future.  Looking at transcript now, it seems pretty clear that was the case; Gates seems to find it important, but not central to the future.  And this difference of opinion, like the new lightweight development models, is fundamental to the discussion of the converging lines between the Web and the enterprise, something I (eponymously) like to call Enterprise Web 2.0.

It's the data, silly 

One key part is that the act of using the Web creates a vast trail of valuable information that is often just as useful as the content on the Web itself.  This "collective intelligence", something John Battelle has been calling the Database of Intentions for a while now, is essentially this:

The aggregate results of every search ever entered, every result list ever tendered, and every path taken as a result. It lives in many places, but three or four places in particular hold a massive amount of this data (ie MSN, Google, and Yahoo). This information represents, in aggregate form, a place holder for the intentions of humankind - a massive database of desires, needs, wants, and likes that can be discovered, supoenaed, archived, tracked, and exploited to all sorts of ends.

In their chat, O'Reilly makes the point that many previously valuable data sources, such as mapping and navigation information, can now just be collected from the online community while they travel.  The commercial software vendors are "really at risk from that kind of true Web 2.0 application that just gathers the data from the people as they go about their lives." 

This of course, is one of the bigger pieces missing inside the enterprise as well. Traditional SOAs and enterprise systems are usually mute about how to enable architectures of participation; they are much more about about connecting the machines and the data, instead of connecting the people inside the machines.  This is the piece that people are finding so valuable about community aspects of the Web 2.0 software experience.  And it's why there seems to be such a grassroots push towards using community Web applications like blogs and wikis to get work done inside the enterprise.

Cheap, agile online software as an aggregate drag on big software 

In any case, Microsoft has been pouring considerable energy and attention in the last six months into their Live Software initiatives and so I was curious to see how much the software giant is picking up on the apparent root causes of the changes in the business.  Stowe Boyd did the same thing earlier today and he came away with a very similar conclusion to mine, that in the final analysis Bill Gates seems to find the infatuation with online versions of core products a little puzzling.

Stowe makes the point however that such a large organization won't be able to compete effectively in aggregate competition with the nimble Web 2.0 "gazelles", which seem to be sprouting up everywhere we look these days.  Stowe says this causes far "too many battles"as customers can increasingly find just the right software to fit their needs and in this world, one-size-fits-all makes less sense.  Still, in a post MIX interview with Martin LaMonica, Gates says that Microsoft has been "pretty rational as the fads roll through," and that seems to capture the spirit of the caution we're seeing.

The somewhat touted ajaxWrite this week does show that potshots at entrenched mainstream products will be taken.  The fact is, however, that we have yet to see anything truly disruptive to the big software vendors on an individual basis (excepting perhaps, Salesforce.com).  Despite this, I predict that the cumulative drag effect of Web 2.0 software will be the real disruptive force in the near term.  That, and the fact that usage patterns haven't really begun to switch.  I might personally be using Writely, Flickr, del.icio.us, online project management tools, and Web mail clients from inside FireFox for most of my daily work, but most other folks haven't gotten there yet.

Data storage the killer app for the Web platform?

Finally, a couple of other interesting points.  One, O'Reilly made much of using data to leverage and support a related hardware platform for greater cumulative effect, like Apple does with iTunes.  Gates didn't respond meaningfully to the concept, yet it's clear that we're seeing a rise in significant attempts to be the key conveyer of data. Google Base and Amazon's amazing S3 are just two examples of online Web storage that is intended to be general purpose enough for everything. The unstated message: Whoever amasses the most data first, wins.

These efforts show that the Web is starting to enter maturity as a true platform, only increasing the pressure on traditional software vendors and customers to compete using these cheaper (due to scale) and nimbler resources. 

And two, Gates admits that Microsoft miscalculated the value of the advertising business model, again essentially their seeming corporate inability to measure the scale and power of the Web and its associated network effects.  Fortunately, I think this is one of the smaller pieces of the online software puzzle and I'm with Fred Wilson that the most viable business model for online software may already exist.

Will better online software solutions leak en masse through the firewall and remake enterprise IT from bottom-up?