Build a 135TB array for $7,384

Build a 135TB array for $7,384

Summary: Is it possible to build protected storage for less than $55/TB? Yes, it is. Here's how.

SHARE:
TOPICS: CXO, Hardware, Storage
13

Is it possible to build protected storage for less than $55/TB? Yes, it is. Here's how.

Storage Pod II Almost 2 years ago in Build a RAID 6 array for $100/TB I showed the 1st storage pod. This version includes what the developer and unlimited backup provider Backblaze has learned from running over 16 petabytes of storage on over 9,000 drives.

The new pod cuts the TB cost in half while increasing performance - the pods can saturate a gigabit Ethernet - mostly due to the hard and under-appreciated work of drive manufacturers. The new pod uses Hitachi 3TB drives to boost density and reduce cost.

Build? Backblaze doesn't sell the pod, so you'll have to build your own. Their blog post includes a detailed parts list. You'll need software expertise as well: they use Debian 5, the ext4 file system and a logical volume manager above the RAID 6 storage but below the file system.

This is a tested-in-production design:

Courtesy Backblaze

Courtesy Backblaze

A major university stores medical images on them, and there are other production users as well.

Lessons learned Backblaze is employee-owned, not venture-backed, and profitable - not least because they've cracked the cost nut on protected storage. Like major cloud providers, they've built their infrastructure on commodity hardware, building resiliency into the software. Fail-soft as it were.

Running such a large number of drives, they've learned a few things.

  • High infant mortality. They've taken to burning in new pods for several days before putting them into service due to infant mortality and the relative difficulty of replacing drives in the middle of the box.
  • 5% annual failure rate. That's across all drives, young and old, of their population of over 9,000 drives. And in 4 years they haven't seen any drives die of old age.
  • Hitachi's HDS5C3030ALA630 drives have a sub-1% failure rate. The latest and greatest from Hitachi has worked for Backblaze.
  • All parts come with a 3 year warranty. That means that other than labor, your maintenance is free for 3 years. Compare that to your BigCo service contract.
  • Heat doesn't correlate with drive failure. Some drives run hotter than others, but Backblaze hasn't seen any increased failure rates because of it. That finding agrees with earlier research.

Costs They offer an interesting figure which shows just how profitable Amazon's S3 storage service is:

Of course, Amazon offers many other services and features beyond a barebones box, but whoa!

The Storage Bits take Backblaze is giving storage-loving peoples of the world a gift by open-sourcing their storage pod design. Most business files are opened only a few times, so why put them on the most costly storage you can buy?

I think Backblaze could have a nice little business selling these online for $25k - still a steal - but they're focussed on offering unlimited backup for less than $4/mo. And I thank them for that too.

Comments welcome, of course. Backblaze is not, sadly, a client.

Topics: CXO, Hardware, Storage

About

Robin Harris has been a computer buff for over 35 years and selling and marketing data storage for over 30 years in companies large and small.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

13 comments
Log in or register to join the discussion
  • RE: Build a 135TB array for $7,384

    How about that cloud?
    Aerowind
  • More designs like this please

    If we had a similar solution for networking fabric then consumers and businesses could telll the likes of Amazon, M$ and other cloud providers what to do with their public offerings.

    There is surely a huge opportunity for a new company to undercut the usual suspects.
    jacksonjohn
    • The start of a private cloud

      @johnfenjackson@...

      While some of the cloud offerings (from Apple, MS, Google and such) have some really nice features, what I really want is a private cloud.

      A little computer not much more than one of the new MacMinis with about 1-2TB storage attached. Included is all the software needed for me to access it just like iCloud or what not.

      That, to me, sounds like a prime business opportunity.
      Bruizer
      • ... in which case how about ...

        http://h71016.www7.hp.com/dstore/ctoBases.asp?ProductLineId=431&FamilyId=3289&jumpid=re_R2515_store/enProdCat/MicroServer/MicroServer

        ... in the UK they have been on offer since launch for c$160!

        I've had a couple happily running Windows 7 in the base configuration of 1GB RAM. HP's smaller servers (unlike DELL's) will accomodate consumer disks OOTB.
        jacksonjohn
      • RE: Build a 135TB array for $7,384

        @Bruizer A Mac Mini would be overkill for a home cloud - try a Tonido Plug instead: http://www.tonidoplug.com/
        blacksmith_tb
  • Prebuilt Backblaze pod

    Recently the company I work for purchased the parts to build our own Backblaze pod. It took a while to get everything ordered, put together, and going. The company that we purchased the case from (Protocase.com) recently sent an email informing us that we could now purchase a FULLY ASSEMBLED unit (minus the drives) for under $5500. We are already discussing purchasing another, fully assembled, pod for offsite replication. (Our current pod had FreeNAS 8.0 on it)

    The IT department here researched almost 10 different solutions for storage of our backups. This is by far the best storage/dollar solution we found.
    dsumner25
  • So what about the Human cost?

    It's funny whenever I see these analyses they always leave out the cost it takes to keep people around who's job it is to keep this stuff running.
    fdumlao
    • RE: So what about Human cost?

      @fdumlao <br>*Disclaimer* I work at Backblaze - Our costs actually include the cost of Sean, who maintains our fleet of over 200 storage pods. You can read more about Sean in the "Lessons Learned" section of the blog post: <a href="http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/" target="_blank" rel="nofollow">http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/</a><br>Sean does get a little help every now and again as each pod weighs about 140 lbs when fully geared up, but typically he's able to handle all 9,000+ drives on his own!
      YevP
  • Is local redundancy enough?

    I think the picture says it all. If there's a fire in one of the racks, how much data will be permanently lost? Can backblaze survive a natural disaster? What about a multi-day outage? RAID6 does not help this scenario.<br><br>I think anyone that is concerned about the durability of their data needs to think carefully about storing their data in a single location. Unfortunately in a centralized storage model where all data is densely stored this is difficult to accomplish. Storing data is multiple data centers doubles (or triples) the cost, and adds new BW costs that can not be ignored?<br><br>What's the alternative -- decentralization. See <a href="http://www.symform.com/resilient-storage-architecture.aspx" target="_blank" rel="nofollow">http://www.symform.com/resilient-storage-architecture.aspx</a>.
    bassamtabbara
    • RE: Build a 135TB array for $7,384

      @bassamtabbara A totally unrelated issue.

      That comes under disaster recovery, not basic costs.

      The costs of decentralization is also less than the competition for the same reason that a single site costs less.

      The storage is just cheaper than what you, a competitor, provide.
      jessepollard
  • RE: Build a 135TB array for $7,384

    These things would be great for any site that has to store large amounts of media e.g. Facebook(photos), WikiMedia commons. It would also make an awesome NAS for my <a href="http://addto.it/Windows-7-Home-Theater-PC-(HTPC)">HTPC</a>.
    jonfleck
  • RE: Build a 135TB array for $7,384

    This is great. But... how safe is the data? Is is protected against data corruption? You know, ECC RAM is needed because bit flips occurs spontaneously. Hard disks are also affected by this problem. See the spec sheet of any Enterprise SAS disk: "1 irrecoverable error on every 10^16 bits read". ECC RAM is needed. The same protection is needed with disks. Have you thought about this? Hardware raid does not give protection. Here are some research papers on this problem, just read the links:
    http://en.wikipedia.org/wiki/ZFS#Data_Integrity
    Orvar
  • HDD failure not changed by heat? Interesting?

    I find the industries comments about heat and HDD failure rates rather interesting. Heat defiantly reduces the life of normal microcircuits and HDDs have microcircuits. Those statements by the industry where they claim no correlation between failure rates and heat may be wishful thinking combined with twisted data.

    I began my carrier of working on computers back in the 70s. Granted it is only anecdotal evidence but I have noticed that my HDD failure rate dropped dramatically when I ensured there was always an empty space between HDDs and when I made sure to always add extra fans for HDD cooling.

    Years ago I noticed that the HDDs that tended to fail were often the HDDs that had the poorest cooling. Example: center HDD in a cluster of 3 or laptop HDD (used about 10 hrs a day) where I noticed the laptop had excessive heat building up and the laptop burned through 2 HDDs in 9 months but after I added a cooling pad and upped the speed of the internal fan the laptops last HDD is still running fine 7 years later.

    Because of past experience I always ensure that HDDs have extra cooling and now I have a very few problems with failing HDDs (knock on wood).

    Again, I realize that my evidence is anecdotal but lets get some reality in things. For a long time now it has been a known fact that heat will reduce the life of microchips and HDDs have microchips. Even if heat does not cause any problems with the mechanical components and even if heat does not increase the degradation of the platter (as if), we do know that heat does reduce the life of at least some of the components on a HDD.
    John238