Researchers to build peer-to-peer storage monsters

What would Napster on steroids look like?
Written by Mary Jo Foley, Senior Contributing Editor on

Maybe like a couple of the futuristic distributed-file-system projects in development by some universities and companies.

Microsoft Research is working on one such beast, a project codenamed Farsite. The University of California at Berkeley -- with backing from the Defense Research Projects Agency, IBM and EMC -- is working on another, codenamed OceanStore.

This week, in fact, the lead developer of Microsoft Farsite is attending a Lake Tahoe research retreat sponsored by UC Berkeley computer science students working on OceanStore.

These research projects are years away from finding their way into commercialised products -- if they ever do.

But the projects do indicate that companies and colleges are anticipating that in the not-too-distant future, bandwidth advances could expand exponentially the size and complexity of applications and services that consumers will expect to run over the Web.

A distributed file system controls where files are stored -- for example, on individual PCs or on central servers.

The central servers in charge of distributed file systems can get overtaxed, however, when too many people use them.

Enter the concept of a "serverless file system" -- a kind of network of clients with no hierarchy, says Microsoft Farsite leader Bill Bolosky.

"We know that [PC] disks aren't full today, and machines are idle most of the time," Bolosky said. "Disks are getting bigger faster than files are. If you could build a centralised file server, it would be more reliable, and it wouldn't require centralised administration."

Napster and other peer-to-peer (P2P) file-sharing software are built on a similar premise but with one key difference, he added.

"Napster does have a central directory. Only its file store is peer-to-peer," he said.

With Farsite -- a concept Bolosky and his five-person team have been dabbling with for the past year -- all elements would be distributed.

As such, Farsite would be based on a true cooperative storage model, rather than on a centralised or local one.

These kinds of mega-file systems aren't for the casual consumer.

Microsoft's target for Farsite, Bolosky said, is a "large company or university, meaning an organisation with around 100,000 machines, storing around 100 billion files, containing around 10,000 terabytes of data".

You read that right: 100,000 computers networked together.

The OceanStore team has set its sights even higher, Bolosky said, describing networks without servers, consisting of 10 billion computers, and containing 100 billion terabytes of data.

OceanStore, as it has been described on various Web sites, is more of a generalized object store than a pure file system.

But as it is being developed by the same Berkeley group that worked on the xFS file system, it has its roots in the distributed file system world.

And like Farsite, OceanStore has implications for mobile computing.

One OceanStore Web site explains the concept this way: "Unlike other file systems, OceanStore provides truly nomadic data that is free to migrate and be replicated anywhere in the world."

Have your say instantly, and see what others have said. Click on the TalkBack button and go to the ZDNet News forum.

Let the editors know what you think in the Mailroom. And read what others have said.

Editorial standards


How to use your phone to diagnose your car's 'check engine' light
BlueDriver Bluetooth dongle

How to use your phone to diagnose your car's 'check engine' light

For $2, you can finally give your Mac an incredibly useful feature Windows has always had

For $2, you can finally give your Mac an incredibly useful feature Windows has always had

Elon Musk drops details about Tesla's humanoid robot

Elon Musk drops details about Tesla's humanoid robot