For Peter Jackson's Weta Digital the Cloud does not compute

Kathy Gruzas, CIO of digital effects production house Weta Digital, explains why the company is sticking with its own datacenter.
Written by Rob O'Neill, Contributor

When Kevin Baillie, co-founder of California-based special effects house Atomic Fiction, took the stage at last year’s AWS re:Invent conference in Las Vegas he made a compelling case for the benefits of cloud computing.

Baillie said Atomic fiction was able to dial up “Pixar-sized” render farms in minutes and faced no physical limits on its computing power by using AWS.

That in turn allowed Atomic Fiction to put its resources into people, not into computing and datacentres.

In an industry with immense and sudden peaks and troughs in computing workloads, cloud seems a no-brainer.

But for Peter Jackson’s Weta Digital, maker of the Lord of the Rings and Hobbit trilogies and of King Kong among other projects, the sums don’t add up.

CIO Kathy Gruzas says while there are many cloud applications Weta could use and which are “theoretically very attractive”, when the company has studied cloud costs they do not stack up.

Weta Digital has invested over many years in its own supercomputing capacity. Its datacentre is powered by 5000 HP servers from the latest Ivy Bridge microarchitecture to older G5 blades.

The datacentre has to scale for peak capacity, not average use, and that’s not cheap. It also means the render farm is not busy all the time, so capacity is not being fully utilised.

Gruzas said Weta has looked at cloud, which could address that problem. The company even performed costings, but a number of factors mean it has yet to be embraced.

Bandwidth is an issue for the Wellington, New Zealand-based company. It’s expensive, but the real issue is the volume of data and latency.

Latency is managed using IBM Aspera’s FASP transfer technology because FTP won’t cut it, said Weta Digital’s systems manager, Matt Cunningham.

Weta can render 50TB of data overnight during production. That is then studied and changes made before it is rendered again.

Fast turnaround is essential to keep up velocity, Gruzas said.

In Sydney, where she used to work, production houses were linked by LAN-speed networks, allowing them to collaborate and share each other’s capacity.

If there was a suitable cloud datacentre in New Zealand accessible at LAN-speed, it would be a no-brainer.

Weta also looked at using Sydney-based Steam Engine, but that didn’t add up either.

But it isn’t just computing power that fails the test. As Weta finishes production, files are archived on tape. The company has at least 8PB of data stored this way.

Last year the company looked at renewing that capacity or finding an alternative. Weta took a close look at Amazon’s Glacier archiving service, which is available through the AWS Sydney datacentre.

“When we costed it up it didn’t make financial sense,” Gruzas said. “We find that with a few cloud services.”

She concedes if you were a “greenfields” site faced with capital investment it might be different.

And that leads to the issue of competition. Weta Digital’s investment in computing capacity has been a "moat' against challengers, but is it still?

Gruzas said cloud will certainly change industry investment patterns in the future but Weta’s direct investment has been a good one. It can also be written off against numerous projects over its five-year lifespan.

“Quite a bit if capability is with size. We have the infrastructure to get things done quickly.”

A similar-sized service to Weta’s in the cloud would cost a lot, she said.

“I would be surprised if you could cost it up for less.”

But there is another, perhaps more important, constraint: the clients who dictate how content is delivered and stored.

Clients mostly dictate delivery over the net but with full encryption and security. Sometimes this is done in partnership with a specialist ISP.

Sometimes encrypted hard drives are shipped by plane as backup.

Data ownership is a particularly sensitive area, Gruzas said. Some clients are not happy about having their expensive intellectual property in the cloud.

“We do what the client tells us and provide options,” Gruzas said.

A lot of the technology is also quite proprietary and sector specific, she said.

Storage, via a is via high performance NAS, is the biggest issue, said Gruzas.

Weta is constantly reevaluating  and testing storage technology.

“It almost makes the cloud a non-issue,” Cunningham said. “If you are having troubles [accessing] internally let alone across the net.”

Gruzas said more standard use cases for the cloud could be considered, especially where there is no client data involved.

Fro Atomic Fiction, though, it is the cloud that makes everything possible. Baillie said 1000 servers for one hour costs the same as 100 servers for 10 hours. that provides computing power and flexibility.

Once production is finished, that can be scaled back to zero servers so unused infrastructure is not being paid for.

Editorial standards