Red shift meets event horizon

I was amazed to read reports of a dinner last week at which top Sun executives sitting around the same table seem to have made statements that totally contradict one another.

On the one hand, Greg Papadopoulos, Sun CTO and executive vice president of R&D, was telling Dan Farber how "his notion of a Red Shift, a move to massive scale computing infrastructure, will become more of a reality in the next year." The notion is linked to the view, expressed in an earlier interview, that the earth's compute resources will resolve into about "five hyperscale, pan-global broadband computing services giants" — with Sun, in its version of this future scenario, the primary supplier of hardware and operations software to those giants. As Papadopoulos reportedly told Dan at last week's dinner:

"Sun is betting on what Papadopoulos termed a 'neutron star collapse of datacenters'. At some point it won't make sense for businesses to build their own datacenters, but to get resources from hosting providers' 'brutal efficiency' for utilization, power, security, service levels and idea-to-deploy time ... Sun wants to provide the infrastructure for the hosters and to be a host itself.

"As an example of the coming Red Shift and massive scale, the Department of Energy's Oak Ridge National Laboratory is building 500,000 square-foot datacenter with 50 megawatts of power, which is enough electricity to power a small city."

Meanwhile, further along the same dinner table, Sun vice president and distinguished engineer Subodh Bapat was telling Nick Carr [update: actually, it wasn't Nick, he just picked up on several reports of the dinner] that "a large data center would suffer a 'massive failure' during the next twelve months, causing 'major national effects', including possible 'national security issues', and underscoring 'the importance of data centers as national assets'." Citing exactly the same Department of Energy project that Papadopoulos had described in glowing terms to Dan Farber, Bapat "warned that 'utilities are going to become a real problem' for such megacenters."

I suppose I shouldn't be surprised that Sun is trying to have it both ways — simultaneously advocating huge consolidation of data centers and then warning what a huge single-point-of-failure risk they represent. After all, that's the way big iron vendors always have behaved. First of all they sell you their grand vision. Then they sell you all the hidden extras required to make the vision actually work in practice — never-ending professional services, huge layers of infrastructure management and absurd levels of duplication to build enough redundancy into the system.

But all this consolidation goes against the grain of the Internet. As Nick says in his write-up:

"The internet was constructed as a relatively simple communications medium, the reliability of which was tied directly to its radical decentralization. No node on the net was particularly important, so the failure of any one node had little effect on the operation of the system as a whole."

Actually that's a slight oversimplification, as the major peering centers are relatively few in number and there are only a few top-level DNS nodes. But the underlying design principle of the Internet — reflecting its genesis in the Cold War years when the threat of all-out nuclear attack was an ever-present reality — is indeed of a resilient, decentralized network that can continue to function even when vast tracts have been laid waste.

Knowing all this, why does Nick go on to swallow Sun's argument — hook, line and sinker? — when he writes:

"[N]ow that the net is becoming a computing grid, delivering complex software services and massive amounts of data, it is also becoming much more centralized. While there's still no single point of failure, the rise of massive data centers serving millions of users means that the failure of a major center could have wide ramifications."

What is it about a computing grid that's inherently "more centralized" in nature? I suppose if you intuitively gravitate towards a centralized utility computing model — as we know from Nick's published writings that he does — then I guess you do tend to see it as inevitable. But such visions always oversimplify the network into a very old-fashioned, central-command pattern that denies the computing power that sits throughout the network, on local servers and on individual clients. While it's true that there are economies of scale to be gained from consolidating certain resources, there are economies of resilience and flexibility that derive from distributing other resources.

In my view, Red Shift is strategically untenable in the modern world. Concentrating the computing grid into a handful of huge data centers exposes a society to an unacceptable risk of terrorist or military attack. Having the grid operated by no more than four to five giant utilities exposes an economy to unacceptable monopoly control. Worst of all, making every citizen dependent on monolithic, faceless utilities is a throwback to the concentrations of power and norms of conformity that characterized the industrial age. They have no place in the information age.

Commenting on the Red Shift debate, Vinnie Mirchandani picks up the phrase 'Petro-Google', an echo of the 1970s term petrodollar. Another phrase that people used a lot in the 1970s was the military-industrial complex. I'm not sure about the petro-Google but I do believe we need to be wary of a dangerous new phenomenon today, which I'll call the techno-utility complex, in which vested economic and political interests conspire together to build huge technology-based utility industries that preserve and reinforce their power bases.

My personal bugbear (bear with me for a paragraph or two, the threads will link up in a moment) is the way the power generation utilities are all falling over each other to build huge wind farms and tidal barriers to generate clean electricity. Why is it that government is so keen to encourage (and often subsidize) wind and wave power generation while solar power technology remains the poor relation in green energy terms?

It turns out that power generation increases geometrically in proportion to the size of a turbine, whereas it's directly proportional in the case of solar panels. So to get the best results from wind or wave power you have to build your wind farms and tidal barrages at industrial scale, whereas any individual can put a solar panel on their roof and start feeding power back into the grid. Even better, solar power at the point of use is more efficient because power isn't wasted in transmission, and it lessens the individual's dependence on the grid. I suspect the real reason solar power gets de-emphasized is because it shifts the power balance between the utilities and the consumer, so no one with any clout is lobbying for it.

In exactly the same way, concentration of compute power in the network is in the interests of those who operate the large data centers. Sun, Google, Microsoft and Amazon will lobby hard for it, but who will argue the counter case, against the techno-utility complex? I'm not saying there's no case for megacenters — they'll benefit consumers where they provide true economies of scale, especially if they're sited next to plentiful supplies of hydro or other carbon-neutral energy sources — but my gut feel is that the ideal solution is a hybrid infrastructure, combining megacenters, transportable midi-centers and client-side resources, each performing tasks they're best suited for and most efficient at.

Knowing the economic power that the utility giants could wield if they were able to scale up to the extent that Sun's Red Shift philosophy envisages, I fear that the case for the alternatives will be argued less powerfully. The danger is that allowing the techno-utility complex to get its way will prove contrary to the true interests of society.