eBay: Data center efficiency = business survival

Back in 2007, the powers-that-be at eBay had an epiphany: The way it was building data centers was unsustainable, especially from a power and cooling standpoint.

"At some point, the cost of running the sites would negatively affect our business," said Dean Nelson, senior director of Global Data Center Services for eBay, during a presentation at the recent Symposium 2010, held by the Uptime Institute in New York.

See also: eBay Sets a Realistic Goal for their New Datacenter

That's when eBay started thinking about its data centers in a very different way, not as something that supports the business but as something that IS the business. When the company has an outage, Nelson says, his boss doesn't ask about the problem in technology terms but rather in terms directly related to the business.

"She doesn't ask how much time we were down, she asks about how much money we lost," he says.

For context, consider that roughly $2,000 worth of goods are trade on eBay every second. Yes, every second. When Nelson spoke in mid-May, he said there were approximately 215 million items listed on the site, a number that grew by roughly 40 million in the past nine months.

So, how can eBay sustain this growth: Nelson says the trick lies in breaking the linear relationship between transaction growth and hardware sprawl. This takes smart design from a hardware standpoint -- servers AND facilities -- AND it takes more intelligent design from an application standpoint. Nelson cares about power consumption because, pure and simple, he is responsible for the power bill.

"The application should tell the infrastructure what exactly to do. It should only consume energy when it is needs to," Nelson says.

This was one of the guiding principles for the company's massive new data center, a $287 million infrastructure investment in Utah that opened in early May and that will house all of eBay's core businesses. Nelson describes the facility as "bullet-proof", not just from a resiliency standpoint but from an efficiency standpoint.

According to Nelson, this particular facility is about 30 percent more efficient than eBay's other data centers at a designed Power Usage Effectiveness (PUE) of 1.4: "We only consume the energy we need, when we need it," Nelson writes in his blog. The entire infrastructure runs at 400V, which delivers 230V directly to the servers. That change alone resulted in a 2 percent efficiency gain, Nelson says.

An interior view of the new Utah data center.

You can read more of the copious juicy details on the facility there. But during his Uptime presentation, Nelson provided other data center managers with five pointers that he believes should drive their own data center overhauls:

1. Create tiered data centers that are rightsized. This actually was a common refrain during Symposium 2010; the notion that many servers are way over-provisioned.
2. Use the latest technology innovation; but remember that improvements in power infrastructure will have a ripple effect on how servers should be configured. And vice versa.
3. Be flexible from a financial standpoint. eBay actuallly move to a two-year leasing policy in order to refresh its technology as often as possible.
4. Look at the total cost of ownership equation in a different way: Transactions per watt or listing, or dollars generated per watt. Power is their biggest operational cost. "Transactions per second really is no longer the primary metric," Nelson says.
5. Drive aggressive technology refresh cycles, which can help you double capacity every 18 months to 24 months within the same footprint. "We have doubled and almost tripled our capacity in some instances and our costs have stayed flat," Nelson says. "That a really good thing."