How the software economy is driven by proprietary work

As should be apparent to readers of ZDNet Talkback forums, I enjoy a good debate. My vision of the ideal forum would be a café table where people discuss controversial issues in an environment where disagreement doesn't imply your opponent is Satan's house cleaner. Richard Stallman seemed willing to play along. Eric Raymond, on the other hand, was the guy at the table trying to take out his opponent's eye with a fork.

Eric recommended that I read The Magic Cauldron, his seminal work on the economics of open source. Though I'd read it before, I did so a long time ago, and certainly not with an eye towards building a response. In truth, this is partly a response and partly a parallel defence of proprietary software that is informed by points made in "The Magic Cauldron".

This instalment presents some foundation theory which details the role played by proprietary software companies in the software economy. Part two begins the rebuttal phase of this series, dealing with the comparative advantages of open source and proprietary software. Part three deals with Raymond's five reasons to choose open source. Part four discusses issues related to developer compensation.

Preamble: Open Source IS Good
I LIKE open source, and think that open-source software is an important component of a modern software economy.

Source code access is a tremendous way to ensure the spread of a particular technology. The right to copy binaries (compiled source code) serves a similar function. Source code access is indispensable in cases where customised variants are required (embedded software is a good example). Source code is conducive to community development, creating a realm within which development load can be borne by many participants.

My difference with people like Richard Stallman is not that I see no value in "free software" (that is, software for which the source code is available, and the right to copy is guaranteed, among other things), but that I think there are benefits to proprietary business models that can outweigh the social costs of charging access to "secrets" in your software. My difference with Eric Raymond, who does see a role for proprietary software, is a question of degree. I believe the benefits to be derived from proprietary software are greater than he does, and that it should constitute a greater share of the market than the 5 percent he proposed in our email exchange.

Proprietary software and "use value"
Raymond makes a distinction between the "use value" and the "sale value" of software: "the use value of a program is its economic value as a tool. The sale value of a program is its value as a saleable commodity."

Raymond notes that most developers don't rely on sale value, directly or indirectly, for income. Rather, the "use value" companies derive from software drives them to pay programmers to build custom products tailored to their business.

Raymond estimates that a mere 5 percent of the market is reliant on sale value. This figure seems a bit low too me, though I can't complain overmuch as he doesn't claim that it is any more than a rough estimate (as Raymond states, "if it were 15 percent or 20 percent, the economic consequences would remain essentially the same"). Assuming the figure is correct, consider the disproportionate effect that presumed 5 percent has on "use value."

Imagine a custom billing system written in Java. The Java runtime is proprietary, even if the source code for the libraries are available for download. A billing system will usually need a database, and unless you choose something like MySQL (unlikely, given that they just included support for stored procedures), the database would be proprietary. Unless you chose Linux, FreeBSD, or another open-source operating system, you would likely build your application atop some proprietary platform.

Future trends seem to point towards the use of more open-source software in the software stack, and it is certainly possible to build something that relies only on "free" software. Even so, a disproportionate share of "use value" in today's software market is created by proprietary software. I don't consider it much of a leap to claim that at least 70 percent of current "use value" is derived from proprietary products.

That's a lot of "bang" for 5 percent of the market.

The source of proprietary "use value"
The roots of innovation spread far and wide. Some innovation appears random, the by-product of "pure research." The notebooks of Ramanujan, a mathematician who managed single-handedly to deduce all of western mathematics from his village in southern India before his invitation to Cambridge shortly before World War I, are still plumbed for new ideas. Ramunajun was a pure ideas person, in that many of his insights lacked at the time any apparent practical application. Topology is another area of mathematics that was considered "pure" by its early practitioners. Today, topology has grown in popularity as modern physics found ways to use it in theories of "Hyperspace".

Most innovation, however, results from attempts to solve real-world problems. Corporations lie at the heart of applied innovation, paying employees to figure out new ways to satisfy "real world demand" as represented by paying customers, either through new ideas or the application of "pure" innovation to customer needs.

Companies, however, are interested in taking market share from competitors. Money is at stake, and competition for it drives companies to differentiate themselves in ways that can create incompatibilities. To critics such as Raymond, this is a Bad Thing â„¢, and it is blamed for the fragmentation that occurred with proprietary Unix.

There are other ways to view this phenomenon, however. The proper manner to adapt technology to general consumer needs is not obvious. This is particularly the case with new technology, as no one has experience of its application. Companies, therefore, will offer implementations that differ in the details, all in an attempt to grow market share and earn revenue.

The motivations, to be sure, are self-interested. Regardless, large-scale parallel experimentation is one of the reasons market systems work better than planned systems. Parallel experimentation in the market for manufactured goods leads to the best means to allocate scarce resources (with "best" defined as the outcome which most satisfies human needs and wants). The same principle applies in software.

Why Proprietary Companies create so much innovation
Proprietary software companies, as noted by Raymond, rely on their ability to convince customers to upgrade to new versions. This is known as the "upgrade treadmill," and it is viewed by Raymond as an unmitigated Bad Thing â„¢.

Viewed a different way, the motivation of vested financial interests is a goad that drives proprietary companies to create new ideas. Given the disproportionate share of use value created by proprietary companies, this is apparently an effective motivator.

Second, proprietary companies are close to the primary source of information regarding human needs: their customers. Companies are best positioned to figure out what customers actually want. Open source, in contrast, is a product mostly reliant on voluntary contributions, and thus doesn't have the same access to these information wellsprings.

Third, proprietary software companies make money to a degree that open-source software companies don't, and it is this extra revenue that enables them to spend more energy on updating software as such. They can use this money to conduct applied research which produces new software ideas, or to conduct proper end-user studies to determine what interests them.

Lastly, developers are paid to orient themselves towards the interests of non-technical users. This is an important advantage, as volunteers are more likely to be interested in writing the kind of software that interests them, while paid workers have a vested interest in teasing out the needs of other people.

Ask yourself why so much open-source software is of the "infrastructure" variety ("plumbing" code, which in no way implies that such code isn't essential), or why Linux had (perhaps still has) such a difficult time making a consumer-friendly operating system. Developers who aren't paid to orient themselves towards the interests of regular consumers will tend to create software that matters to people like themselves.

Where open source makes most sense
Open source is most useful in areas of technology that are well understood. Proprietary companies have the advantage in new technology realms due to better information regarding real-world customer needs and their ability to experiment in parallel regarding the satisfaction of those needs.

Open source also makes most sense in "infrastructure." MOST contributors to open source do so free of charge (though some don't, a point I discuss in future instalments). This means that their work will tend to be oriented towards areas of interest to themselves. Infrastructure is of necessity oriented towards technical personnel, and thus the interests of the open-source contributors and end users (technical personnel) align closely.

The open-source market seems to have fallen into this pattern. Apache is the most popular Web server in existence. HTTP, however, is not a fast-evolving technology domain. The kinds of things required of HTTP are already well understood. Furthermore, the Apache Web server is targeted towards programmers and technicians. Within such an environment, the spontaneous contribution model offers almost as much as the parallel experimentation model.

Partial Agreement
Unlike Richard Stallman, Eric Raymond makes room for proprietary software in his vision of the future. As noted near the end of The Magic Cauldron:

"Applications, on the other hand, will have the most tendencies to remain closed. There will be circumstances under which the use value of an undisclosed algorithm or technology will be high enough (and the costs associated with unreliability will be low enough, and the risks associated with a supplier monopoly sufficiently tolerable) that consumers will continue to pay for close software" (NOTE: I'll deal with the points made in brackets in later instalments).

Eric used Id Software, maker of game software, as an example of a valid reason to release software under a proprietary licence. When it was first released, Doom was in Raymond's words utterly unique. It managed to create a 3-D universe on the low-powered processors of 1993 in ways that remained unfathomable for a number of months. Later, they released the source for the core rendering engine once the techniques used became well understood (among other reasons).

Doom was such interesting technology that it was right for "Id Software" to attempt to recoup its investment through proprietary licensing. Once the technology entered the realm of common knowledge, however, Id released the source code in order to benefit from the network effects and community improvements made possible by open source.

Some Parting Thoughts
Open source would rule the world if we'd reach a plateau in terms of the kinds of things we can do with software. At that point, software would cease to need much in the way of innovation, and the slow pace would leave plenty of time for most of the software world to be standardised by international standards bodies.

This simply isn't the case today. When I look towards the future, I see whole new categories of software bursting forth in response to new devices and new needs. In my opinion, developers have seen only the tip of a software iceberg.

Open source will continue to progress, growing into areas of technology as they become better understood. In areas of fast growth, proprietary software companies will continue to dominate and generate a disproportionate share of use value. This will be the case so long as proprietary companies, with their combination of close contact with real-world customers, stable sources of revenue and incentives to produce new ideas, proves the best way to tease out the intersection between technology and the general public.

John Carroll is a software engineer. He specialises in the design and development of distributed systems using Java and .Net. He is also the founder of Turtleneck Software.