RAMCloud puts everything in DRAM

RAMCloud puts everything in DRAM

Summary: Sometimes in the midst of the endless tweaking needed to maximize storage performance one just wants to say "screw it! Put everything in RAM!" And that's just what RAMCloud does.

SHARE:

RAMCloud is a research paper (pdf) and an open software project. The goal is enterprise-class availability with every bit of active data stored in DRAM, not disk or flash, for maximum performance.

It's the brainchild of John Ousterhout, a Stanford prof who invented Tcl back in the 80s at Berkeley.

Isn't DRAM volatile and costly? Right on both counts, grasshopper, so RAMCloud isn't a 1 for 1 disk-style architecture. No Google FS-style triple replication here, or RAID-style erasure coding.

Instead RAMCloud uses buffered logging:

. . . a single copy of each object is stored in DRAM of a primary server and copies are kept on the disks of two or more backup servers; each server acts as both primary and backup. However, the disk copies are not updated synchronously during write operations. Instead, the primary server updates its DRAM and forwards log entries to the backup servers, where they are stored temporarily in DRAM.

Instead of working around crashes - using multiple object copies - as scale-out storage does - RAMCloud recovers lost data from the DRAM logs to a new server at high speed. Which is possible because all the log data is in DRAM.

Still, total system write throughput is limited by the disk write speed, whose limits are a key reason people are moving from disks. They are looking at using flash drives for this issue, but other techniques, such as log truncation and sharding make it possible to get good performance from several thousand SATA drives.

Lights out! Power failures wipe all the data in DRAM. The obvious defense is to avoid failures: combine battery backup with diesel generator sets. Normal system power ride-through will handle interruptions into the hundreds of milliseconds.

But who is going to trust that? That's why future commercial implementations will insist on logging to stable storage, such as the flash SSDs.

They're getting cheaper fast - faster than DRAM - which will make this a common approach.

The Storage Bits take Make no mistake: RAMCloud is a research project, not a commercial product, years and million$ away from commercial application. But the concept is promising.

Imagine a world where data layout doesn't matter, where apps are optimized for sub-millisecond storage, where 100 byte I/Os are faster and just as efficient as 8KB I/Os. The architectural implications are huge and would take a decade or more to get our heads around.

RAMCloud would also raise the always thorny issue of data tiering: getting hot data on the hot storage and everything else off to disk. There are OK answers for tiering but nothing insanely great.

RAMCloud show's we're far from the end of the line in what storage can do. Faster, better, cheaper: 2 out of 3 ain't bad.

Comments welcome, of course. If you get RAMCloud running on plug computers let me know how it works. Update: The case for RAMCloud (pdf) is more accessible than the academic work cited above. And note that the co-founder of VMware is one of the authors.

Topics: Hardware, Processors, Storage

About

Robin Harris has been a computer buff for over 35 years and selling and marketing data storage for over 30 years in companies large and small.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

3 comments
Log in or register to join the discussion
  • RE: RAMCloud puts everything in DRAM

    John Ousterhout will be giving a public tech talk on RAMCloud at LinkedIn HQ in Mountain View, CA on Wed Oct 12th. Details at http://events.linkedin.com/RAMCloud-Scalable-High-Performance/pub/793368

    This event is free and open to all.
    kapsee
  • Isn't this REDIS?

    Seems to me that this is already out there. You must have some killer feature that makes it better? I don't see it though?
    rbucker
    • RE: RAMCloud puts everything in DRAM

      @rbucker Reddis, like RAMCloud, is a key-value store. The difference is that RAMCloud is designed to run on fast media and has a number of features optimized for DRAM and cluster use. Fast recovery from a server crash is one of the most important.
      R Harris