IT's new battlegrounds in the cloud revolution

As the cloud market matures, effects of scale lock the industry into dependence on a few large companies. In this latest industrial revolution, new entrants will face a tough fight.
Written by Jack Clark, Contributor

Cloud computing is turning into a true utility, as essential to any business as water or electricity. This evolution is driven by massive investments by cloud companies, itself funded from businesses' increased reliance and spending on cloud services over their traditional, internal IT. 

How this shift plays out will determine the fates of many of the world's leading technology companies - some stand to benefit and some stand to lose - and will fundamentally change how businesses are structured. It will also create a new and important commercial landscape. Economists have a term for this: the cloud is industrialising. 

The cloud now has a life of its own. As the public cloud grows, so to do the datacentres of service providers, web hosts, colocation facilities like Equinix and thoroughbred cloud providers like Amazon Web Services, Google or Microsoft.

Last year, Amazon spent around $1.8bn on expanding its datacentres. Google is thought to have spent much more. Since 2007, Equinix, a major co-location provider, has spent $3.3bn on datacentres and now operates over 90 datacentres in 30 markets. In 2012 alone it opened 13 new datacentres across the world. 

Last year, Amazon spent around $1.8bn on expanding its datacentres. Google is thought to have spent much more.

Besides spending vast sums of money on infrastructure, these companies are also becoming hefty consumers of electricity. A 2007 report by the USA's Environmental Protection Agency (EPA) found (PDF) that US datacentres were consuming 1.5% of the country's electricity consumption, and projected that their power use would double by 2011. 

By example, co-location specialist TelecityGroup has a spread of datacentres across Europe that consume 135MW between them. A megawatt is sufficient to support between 750 and 1,000 family homes in developed economies, so TelecityGroup's power consumption is equivalent to a large town or small city. Meanwhile, Google's worldwide fleet of datacentres draw around 260 million watts. 

These figures illustrate the scale at which the cloud industry is beginning to operate. This growing electricity consumption combined with increasingly aggressive expansion plans represents "a massive industrialisation effort" by companies in the technology industry, according to Jason Hoffman, the founder of cloud infrastructure company Joyent.

Forces behind the industrial cloud

The cloud market is industrialising because the scale of the problems the cloud operators deal with demands significant investment. To crack fundamental problems in search (Google), social (Facebook), productivity (Microsoft), or cloud retail (Amazon), - these companies must build top-of-the-line datacentres and do fundamental work in computer science.

With each bout of investment, these companies raise the barriers of entry to smaller companies, while taking advantage of increasingly powerful economies of scale.

Google, for instance, developed the fundamental technology behind the Hadoop data analysis platform to help it crunch large amounts of data. It did this in the early 2000s, yet Hadoop is only now being adopted by other companies.

Similarly its globally distributed database, Spanner, solved some difficult problems and is considered by the database industry to be hugely advanced yet it, also, has been running in production for a number of years. The company employs nearly 500 people in its own skunkworks research organisation and the membership reads like a who's who of significant figures in computer science. This is true industrial research, even though the standard lab equipment is a screen, not a spectroscope. 

Like any effective industrial research, cloud companies depend on innovation to develop technologies that put them significantly ahead of the competition. Though some of the technologies trickle down and become open source - Hadoop being one of the most famous examples - by the time they become freely available to the open source community the originating company has moved on to a more advanced, proprietary system, putting open source a few years behind the incumbents. 

At the same time, this investment puts more space between what a company can do with its own IT and what it can get, normally at lower cost, from the cloud. This classic combination of greater efficiency with greater productivity is an accelerant for enterprise migration. 

"The biggest part of cloud in terms of cost savings is around the total cost of ownership," Lane Patterson, the chief technology officer of colocation provider Equinix, says. "Not having to buy hardware yourself, not having to buy expensive maintenance contracts, not having to buy operating systems and rent them... there's so many levels of cost that get moved out of the equation."

Is IT a cost centre or a competitive advantage?

According to Hoffman, these cost benefits conspire to boil the cloud proposition down to a single question: is your IT a cost centre or a competitive advantage?

Cost centres are  good candidates for the cloud due to the potential to save money, while IT as a competitive advantage happens mostly in highly-specialised industry (high-frequency trading, code breaking, defence), or, more often, within the cloud companies themselves.  

"Not having to buy hardware yourself, not having to buy expensive maintenance contracts, not having to buy operating systems and rent them... there's so many levels of cost that get moved out of the equation," Lane Patterson, CTO, Equinix

There are many more companies for whom IT is a cost centre, than those for whom it is a real competitive advantage. 

This year analyst firms expect companies will spend between $40bn (IDC) and $109bn (Gartner) on public cloud services, rising to $100bn (IDC) or $207bn (Gartner) by 2016. Moreover, it moves company spending on IT from capital to operational expenditure. Opex over capex is an attractive proposition for companies looking to control their outgoings with greater precision and flexibility. 

A natural consequence for cost savings like this are cuts in on-premise budget, as Gartner found recently in its 'cannibal cloud' report, giving the cloud companies the cash to grow while starving their competition.

But while this impacts traditional IT providers, it also presents an unwelcome landscape to new entrants, naturally creating the ideal environment for a few cloud giants and little else. 

Economics makes cloud titans inevitable

Multiply the capex-to-opex business transaction several thousand times and you come up with an economic system that favours large-scale providers selling specialised services to a sea of increasingly capital-light companies.

This creates a very competitive market where price cuts become a key driver to provider advantage. 

Amazon Web Services, for example, has made 21 price cuts to its mainstay storage and compute services since 2006. Each time Amazon cuts its prices, Google and Microsoft tend to cut theirs as well to maintain competitive parity.

The benefits cloud providers gain from their scale make competition by smaller companies difficult. To compete with Google, Microsoft, Amazon, Rackspace, Joyent or other cloud providers, the would-be competitor needs to own or share a a global datacentre presence,  and a reliable software system with some unique advantage - this is a frighteningly expensive thing for a startup to do. 

In the same way it’s rare  to hear of a startup telecommunications company (excluding Mobile Virtual Network Operators (MVNOs)) or energy provider, the economic realities of large-scale cloud computing make it nigh-on impossible for new, small companies to take on incumbents.

"It does become a bit of a scale game," John Engates, the CTO of cloud hoster Rackspace, says. "It is very expensive, it is very costly." By example, Engates say that to even begin contemplating a cloud service you need to have a datacentre of 100,000 square feet or more. This is expensive: Facebooks 300,000 square-foot datacentre in Prineville, Oregon, cost the company $200 million.  

Cloud consolidation

Combined with raising the barriers to entry, this process of industrialisation is likely to breed consolidation among cloud providers. 

Already, some businesses haven’t been able to stand the competitive pressure and have left the market - GoDaddy, for instance, shutdown its cloud service a year after launching it. 

"By 2015, low-cost cloud services will cannibalize up to 15 percent of top outsourcing players' revenue, and more than 20 percent of large IT outsourcers not investing enough in industrialisation and value-added services will disappear through merger and acquisition," Gartner predicted in late-October.

Other businesses have sought out partnerships as a way to benefit from the cloud without being threatened by it, such as cloud software expert Eucalyptus which has hitched itself to the Amazon Web Services cloud to take advantage of its explosive growth. 

The rest of the industry, meanwhile, has banded together to reduce development costs of cloud software in a bid to have enough scale to compete with the major clouds' technology. Over a hundred major IT companies are attempting to create a non-proprietary alternative cloud software stack to Amazon, Google or Microsoft via OpenStack. However, the major operators of the technology are all large capital-intensive companies such as Rackspace and HP and the software is relatively immature. 

All the indicators point to a market which will increasingly be dominated by a few large providers. In the same way that other markets have industrialised and consolidated, it seems likely that the same thing will happen with cloud computing.

This is the first stage of the industry transforming to a utility and it is likely that some companies will not be able to stomach the investment cadence needed to make the transformation. 

Read on to find out how the close the cloud is to becoming a utility. Then discover which companies are likely to win out in the new cloudy world and which may have problems.

Editorial standards