How auto insurer Progressive collected 10 billion miles of driving data from its customers

How auto insurer Progressive collected 10 billion miles of driving data from its customers

Summary: Progressive has quietly amassed a trove of driver data with Snapshot, its opt-in driving tracker device.

Flo and Snapshot. Via Progressive.

The auto insurance company Progressive is known for its quirky commercials helmed by the caricatured sales agent Flo.

But buried beneath the cheap commercial laughs and the drab exterior of insurance policies, Progressive is quietly amassing a trove of driver data — 10 billion miles of driving data, to be exact — with Snapshot, its opt-in driving tracker device.

Progressive began to offer Snapshot to drivers in 2008, right at the birth of the Big Data boom, marketing the device to customers as a means to lower their insurance premiums. Since then, use of the device has been approved in 45 states.

The device plugs into a vehicle's onboard diagnostic port and collects speed at one-second intervals. At the end of every trip, the collected data is sent wirelessly back to Progressive via AT&T's network. Progressive says it plugs the speed readings into algorithms to calculate mileage and hard braking events, with the theory that drivers who exhibit safe driving habits can lower the price of their premiums.

But back to Big Data...

Progressive's usage-based insurance leadership team, which included Dave Pratt, Bill Everett and James Jiang, held an interactive chat session recently to give an inside look at the Snapshot program, how it works, and how they’ve leveraged Big Data as a result.

It turns out Progressive's tools for harnessing and analyzing Big Data run the gamut from refined (version 2.4.1 of Hadoop, R and the full spectrum of SAS software) to bare bones (Excel). The team said Hadoop and SAS is more widely used by Progressive, while the more complex analytical language R is used by only a small number of its internal analysts.

Special Feature

Going Deep on Big Data

Going Deep on Big Data

Big data is transitioning from one of the most hyped and anticipated tech trends of recent years into one of the biggest challenges that IT is now trying to wrestle and harness. We examine the technologies and best practices for taking advantage of big data and provide a look at organizations that are putting it to good use.

"Hadoop works well for massive amounts of data, and it allows us to process the data via multiple processors at the same time," Everett said.

Currently, Progressive collects only two data elements with Snapshot: time and speed. According to Everett, Progressive is beginning efforts to collect GPS data as well.

"Our current product does not include location data. We're currently collecting GPS data from a sample of customers for research purposes only," he said.

And while Progressive uses AT&T's wireless network to transmit data from the Snapshot device back to Progressive's data warehouse, the process is not done in real time. But at some point, mobile phones could be used for that purpose, Pratt said. 

"We're currently testing mobile phones as an alternative data gathering option," he said. "If mobile proves to be accurate, we might be able to offer new, real-time services."

Snapshot collects up to six months of data from drivers for discount purposes, and the data is archived for years as per the insurer's regulatory protocol. The drivers who opt in for the Snapshot program have online access to the data they generate at the end of each trip, using their Progressive policy account to get visualized reports.

No intention of sharing its data 

So, with all of that data building up, one could wonder if Progressive has ever offered — or considered offering — analysis of the data to external entities. 

After all, there are a number of ways that data could be used beyond lowering insurance rates. For example, planners of urban infrastructure could use the data to determine which roads are the most problematic for drivers and in need of work or repair.

But Progressive has no intention of sharing its data trove, according to Pratt.

"It's very important that we keep individual customers' data secure," he said. "Even if it doesn't include name or address, we need to avoid any disclosure of individual information."

Looking ahead to the maturation of smart automobiles, Progressive is hopeful that its data collection will not only continue to remain relevant, but also become easier and more efficient.

"We hope that new technology (whether OEM telematics or mobile phones) will make Snapshot even easier for our customers," Pratt said. "We can use the same analytic techniques on data from new sources."

Topics: Big Data, Enterprise Software, Innovation

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • There is atleast one company has ethics

    There is atleast one company has ethics, thanks for not selling. I will continue using progressive over others.
    • RE: There is atleast one company has ethics... for now

      BUT, what about in the future, when a new CEO, driven by the need to 'increase shareholder value' decides to change the policy?

      ONCE they have the data, HOW can you be sure it will not be used in counter-productive ways?

      The best solution is NOT for them to have that data AT ALL.

      BTW, that's nearly 40 years behind the wheel, and dealing with a few insurance companies along the way's worth of experience.
      • It'll be hard to counteract though

        I totally agree with both posters. Good for Progressive to committing to keep user data private....for now.

        However, if Progressive and any other insurance company dangles the carrot of lower rates and their customers have the option to knowingly opt in it's going to be tough for a lot of people to say no to that deal.
  • On the PLUS side...

    This allows good drivers to directly benefit from their driving habits, as opposed to sharing the burden of poor drivers costs due to multiple collisions and reckless driving habits.

    This is why Credit ratings are a good thing in considering ones personal history as to their trustworthiness.

    The ONLY people who won't like this....are the ones who have a bad record/history and don't want to take responsibility for their own actions/choices.

    Good for Progressive and good for the responsible drivers/customers.
    • yes and no.

      The data is only a small part of the picture. on OBD data I am a bad driver. when a light turns green I accelerate hard to get though the light so as many people behind me get though as well as possible (better for pollution too if dozens of cars are not stuck idling at lights that don't have to be.)

      So I would not be one that showed up as "good habits" by the stats, but I have had 1 accident in 25 years of driving and that was when I got T boned by a holden that ran a stop sign and slammed into the side of me.

      I'd have been able to avoid that one too, but I couldn't swerve because their were cars coming the other way and I'd have missed him but then had a head on with one of them.

      So the stats on driving would make them think I was a bad driver, but my stats on accidents are perfect.. because OBD does't have any way of detecting knowledge of road rules, spatial awareness, car maintenance, driver attention etc etc.. it's only a tiny part of the picture.
      • Are you really a bad driver, frankieh?

        Unless you're peeling rubber away from that light, that hard acceleration isn't going to be enough to identify you as a risk. It's the hard decelerations that Progressive is more likely looking for - - skids to a stop to (barely) avoid hitting the car in front that you've been tailgating. Now, that's the risky behavior that they're hoping their Big Data analytics will reveal. And, probably successfully.
    • Incorrect

      "The ONLY people who won't like this....are the ones who have a bad record/history and don't want to take responsibility for their own actions/choices."

      I have a good driving record and I'm not sanguine with the idea of large corporations playing backseat driver every time I use my vehicle. I happen to take my privacy very serious.

      You are a much more trusting person than I.
  • talk about giving the keys to the devil

    I am dumbfounded anyone would agree to let an insurance company monitor them. Insurance companies are not ethical entities, and are constantly trying to justify higher rates and not paying claims. Just trying to provide a good service and growing a customer base based on that service like other businesses is just not in the insurance companies' vocabulary. The fact they are not selling their data does not impress me. Like they were going to give away their proprietary data anyway! Anyone who agrees to this must have de-evolved before the Neanderthal days.
    • I'm not dumbfounded

      Just as in Lake Wobegon, where every student is above average, the people who gladly put a Snapshot in their OBD port are the one who genuinely believe that they are about in the 90%ile range of "good drivers", and so they are gleefully seeking the discount that Progressive will offer them. Or, so they thought.
  • Big Data Brilliance!

    Let's connect some dots here:
    If an insurance company determines their average annual spend on claims
    ….and that spend is spread across their entire client base
    .…then would it not be beneficial to determine who their high-risk customers are
    ….and find a way via pricing and ‘personal choice’ to cherry-pick the low-risk customers
    ….and send the high-risk ones to their competitors to deal with.
    Even better if they can get the information up-front and prevent high-risk customers from coming onboard in the first place.
    And you bet they have no intention of sharing that data
    • Youi seem to do that now.

      As soon as I told them my car was an XR6T (Turbo falcon) they priced themselves above everyone else, because they don't want my business.. despite my perfect record.
  • Also lets not forget the "old" data either.....

    Tickets, warnings, points, suspensions, retaking driver’s tests, DUI/DWI and yes even your credit score. Take the old historical data and the new behavioral probability info and they've stacked the deck statistically-speaking significantly in their favor....which enables them to keep low-risk, financially stable customers and offer them lower rates...all while advertising those lower rates nationwide and even comparing competitors "for your convenience".
  • Huh?

    "But Progressive has no intention of sharing its data trove"

    For-profit companies can always be trusted, that's something you can take to their bank.
  • and the data is archived for years

    Or, perhaps, to be more grammatically correct, "...and the data are archived for years"