The disk error mystery

The disk error mystery

Summary: You'd think that after 50 years and many billions sold, disk drives would be well understood. And you'd be wrong. Take the case of latent sector errors.

TOPICS: Hardware, Google

You'd think that after 50 years and many billions sold, disk drives would be well understood. And you'd be wrong. Take the case of the outer-track errors.

Thanks to zoned bit recording the bit density of each track is roughly constant across the disk. But more errors occur in the outer tracks - and on some drives on the inner tracks too. What could be going on?

Latent sector errors (LSE) are errors that are undetected until you try to read the data and then the drive says "oops!" In the landmark study An Analysis of Latent Sector Errors in Disk Drives (pdf), researchers found that 8.5% of all nearline disks like the ones most of us consumers user are affected by latent sector errors.

If, like me, you use more than 10 drives, you probably have a drive with LSE. Maybe even two. And if a drive has 1 LSE, it is much more likely to have others.

Deep dive In a deeper analysis of the same data, Understanding latent sector errors and how to protect against them researchers found an interesting anomaly:

The first part of the drive shows a clearly higher concentration of errors than the remainder of the drive. Depending on the model, between 20% and 50% of all errors are located in the first 10% of the drive’s logical sector space. Similarly, for some models the end of the drive has a higher concentration.

Here's the graph from the paper:

Error location on various drive models. Capital letters denote SATA drives, lowercase SAS & FC drives.

Error location on various drive models. Capital letters denote SATA drives, lowercase SAS & FC drives.

Now why is that? In Understanding the authors wonder:

We speculate the areas of the drive with an increased concentration of errors might be are areas with different usage patterns, e.g. filesystems often store metadata at the beginning of the drive.

Sounds reasonable. But later they note:

In particular, a possible explanation . . . might be that these areas see a higher utilization. . . . [But in other research at Google there was no] correlation between either the number of reads or the number of writes that a drive sees (as reported by the drive’s SMART parameters) and the number of LSEs it develops.

Which is it? Disk drives are busy boxes. Possible explanations are:

  • Poor data. Maybe the Google data isn't fine-grained enough to discern workload-related LSEs.
  • Wobbly outer tracks. Lower block numbers usually map to outer tracks where linear velocity is highest. Rotational vibration might cause LSE to cluster on outer tracks.
  • Start/stops. Spinning up a drive is wearing: cold bearings; motor stress; maybe even head wear until fly-height is reached.
  • Lube migration. Disk platters are lubricated to keep them smooth and to minimize wear. This layer can migrate to the outer tracks over time, where it would increase head fly height, making bits harder to read.

The Storage Bits take At present the mystery remains. But the implications for RAID arrays are important.

The 1988 RAID paper assumed that disk drive failures and errors are uncorrelated. But we now know that isn't correct.

Disk failures tend to occur together. LSE - which can kill a RAID 5 recovery - also tend to cluster on particular drives, particular places on drives, and at particular times. Failures are way more correlated that we suspected 20 years ago.

Few desktops should use RAID. If you do use an external SATA RAID 5, make sure you have a reliable backup because chances are good you'll need it. RAID 6 is the way to go when using large SATA drives.

Modern disk drives are amazing precision devices that make fine Swiss watches look as delicate as a strip mine in comparison. Yet we don't understand everything about them.

Kudos to companies like NetApp who support research into disk behavior. As more of the world's data resides on disks, the more important this research becomes.

Comments welcome, of course.

Topics: Hardware, Google

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • Firmware on disk

    The think that is hurting consumer the most nowadays is that disks read the firmware of the disk platter. Done to lower costs but makes it far more difficult to salvage data.
    • Sorry, none of that makes sense.

      Are you saying that a machine's BIOS is located on disk? It's not - the BIOS is on the motherboard - take your HDD's out of your PC and see if you can still get to the BIOS. Hint: You can.
      • He is not talking about the bios...

        He is talking about the controller code on the disk itself.
      • hard drive firmware

        The hard drive has two types of firmware. A bootstrap loader which resides on the controller (call it equivalent to the BIOS) and runtime firmware which is loaded from a reserved area on the hard drive itself (call it the operating system). The controller executes the bootstrap during startup to load the runtime code. It's cheaper than adding flash memory to the controller but if the hard drive has issues in the firmware storage area, you're out of luck. Main reason why the old trick of swapping controller cards when attempting to recover data from a hard drive is very little use on newer hard drives.
    • Thanks for the heads up. I was not aware of this.

      If that is the case now, at least for consumer drives, I can see where data salvage is going to be difficult, if not impossible. This makes data backup even more crucial.
    • This would kill the disk

      This kind of error would cause a whole disk failure. That's not what this article is talking about, and I don't recall hearing about this as a major issue.

      Besides, wouldn't the drive store multiple copies of its firmware? That's how defect maps used to be stored way back when.
  • Eliminate the outer 7% of the platter to increase reliability?

    For some people reliability may take precedence over storage space. I wonder if it would be beneficial to take a 7% hit (5% to 10%) on the storage space to reduce potential errors? It does not seem as if it would be difficult to map a HDD so that the outer 7% or so of each platter was blocked out as unusable.
    • Done years ago . . .


      Vendors started shipping 2.5" platters inside 3.5" years ago. At 15k
      RPM aluminum isn't stiff enough to handle 3.5". There were some
      glass platters on some fast drives for a while, but I think the industry
      stopped using them several years ago.

      Part of the vendor problem is that it is easier to sell capacity to
      civilians than "fewer errors". The first company to do what you suggest
      would be hurt - which is part of the reason for an industry-wide
      migration to the 2.5" form factor in the next 5 years.

      R Harris
      • Aluminum platters...

        are stiff enough, and were used decades ago to store data. In the 80s, big CDC hard disks with removable aluminum platters and flying heads were the standard. However, in case of a head crash, the drives generated hundreds of aluminum and ferrite shavings. It took the techs days to clean shavings out of the drives.

        That's why sealed "Winchester" hard disks replaced CDC drives. Modern sealed hard disks are the successors to Winchester drives.
        • Yes but

          As I recall the platters were about a quarter of an inch thick, and the spindle was about six inches across.

          Those old EDUs were industrial grade tools built so survive constant yanking in & out of drives by not particularly careful or gentle computer operators.
        • Tolerances were a lot greater back then

          I don't think early drives spun at 7200+ rpm. And tolerances were much, much greater back then. A little wobble to the platter could be tolerated. These days when platters have tens of thousands of tracks on them, it's a bit more difficult.
      • I was thinking, aftermarket and being implemented by the consumer

        I was thinking, aftermarket and being implemented by the consumer rather than being implemented by any specific vendor.

        Based on the article it would seem that a consumer who was concerned more with data integrity than storage space could get a utility that would allow the consumer to block out the outer 7% of any HDD. This would slightly reduce the storage capacity but it would theoretically increase reliability.

        I would not expect manufacturers to block out usable storage space on HDDs but some consumers may consider doing just that.
  • The solution is around the corner - SSD

    In 10 years we will be thinking of rotating magnetic storage like we now think of magnetic core memory.
    • Maybe, but I'm not convinced.

      Not only are disks cheap, but they also handle certain workloads very well
      - better than flash. Phase change memory may alter the landscape, but I
      think Jim Gray put it very well when he said disk is the new tape and flash
      is the new disk. Tape is still in wide use in the enterprise and archive
      quality disks would be popular with consumers - if vendors build them.

      R Harris
      • Me neither

        There's a limit to how many times you can change the state of any address in a flash memory. No big deal for memory in cameras & mp3 players etc. But for a business server handling tens of millions of writes a day its a non-starter.
  • RE: The disk error mystery

    The graph looks a lot like a hysteresis loop.
    • Please, elucidate!

      I'd like to know more.

      R Harris
      • Googlefied
    • Magnetic field a property of the disk shape?

      If the hysteresis shape is indicative of the cause of magnetic head failure at the disk edges, then maybe we should be designing a media in a shape other than a disk, and instead of spinning the media, the head should be the moving part, something like a (head) laser emanating from a point source inside a spherical media and writing on the inside surface.
      • that would be cool...

        a tube drive... and probably quieter too. probably use less power as it would take as much force to turn. Maybe you should patent it.