Lies, damned lies, and Android benchmarks

Lies, damned lies, and Android benchmarks

Summary: It's not just Samsung, AnandTech has found that almost all Android vendors are playing fast and loose with benchmarks.

SHARE:

I've been making, evaluating, and using benchmarks for decades. Along the way I've seen vendor after vendor try to play games with them so that their products would look better than the competition. So, I wasn't surprised when the news broke that Samsung had fudged recent Galaxy Note 3 benchmarks to make its hot new smartphone look better. I'm even less surprised that AnandTech has subsequently discovered that almost all Android device OEMs have rigged their devices to show their best on certain benchmarks.

htc-one-antutu-cpu-v1 benchmark results
Benchmarks can be helpful in making buying decisions, but since they're so often gamed, it's wise to never put too much trust into them.

AnandTech, which has specialized in hardcore technology reviews since the 1990s, found that, "With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device" that checks for specific benchmark tests. When such a device finds that it's being tested, it automatically drives its "CPU voltage/frequency to their highest state right away."

The other exception to the rule that everyone cheats was that none of the various Nexus devices showed signs of tampering. That's because these devices use Google's stock Android Open Source Project (AOSP) for their operating system and "the optimization isn't a part of AOSP."

In each case, it's not that the device was feeding the benchmark fake numbers. Instead, as the devices detected that a benchmark was being used they'd simply turn on all their cores and run at the fastest possible speed. The result was that, for the purposes of performance tests, they were showing their best possible results rather the results they'd show under normal use.

AnandTech also found that each OEM optimized their devices for different benchmarks. Only in one case, Qualcomm's Vellamo Mobile Benchmark, did AnandTech find that all vendors' flagship devices "seem to game this test, which sort of makes the point of the optimization moot" since when everyone is cheating the field is, ironically enough, back to being level.

The smartphone and tablet OEMs, and not the chip manufacturers, seem to be the ones behind this latest round of benchmark tampering. AnandTech wrote, "I know internally Intel is quite opposed to the practice (as I’m assuming Qualcomm is as well), making this an OEM level decision and not something advocated by the chipmakers."

The benchmark vendors have also been long aware that vendors and OEMs will try to game their tests. Some, such as FutureMark, make a point of stating that trying to detect their benchmarks take advantage of the test is not allowed.

While the benchmarking companies and publications strive to make testing transparent and above board, the vendors are constantly trying to put their best tech foot forward by any means they can. So, as Michael J. Miller, CIO of private investment firm Ziff Brothers Investments and former editor-in-chief of PC Magazine, pointed out in 2011, "Every benchmark needs to be taken with a grain of salt."

AnandTech also noted that, "The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0-5 percent, and somewhere south of 10 percent on our GPU benchmarks as far as we can tell. I can't stress enough that it would be far less painful for the OEMs to just stop this nonsense and instead demand better performance/power efficiency from their silicon vendors."

There's nothing new about any of this. I've been in the benchmarking game for almost 30-years now and while AnandTech hopes that the OEMs will see the error of their ways and stop trying to game benchmarks I can say with perfect confidence that they won't.

People want a magic number that will tell them with a glance if they're buying the best PC, tablet, smartphone, car, what have you. For computing devices that magic number is a benchmark number. So long as people want a quick, one-number answer, testers, benchmark makers, and vendors will continue to play a tug of war over benchmark results.

As I tell anyone looking to buy any technology, look beyond the benchmarks and to the entire package to decide if it's the right device for you. There are no magic, absolute numbers that can tell you if something is the right buy for you. Only you, after looking over benchmarks, reviews, and your own hands-on experience, can decide what tech is really right for you.

Related stories:

Topics: Mobility, Android, Hardware, Smartphones, Software, Tablets

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

36 comments
Log in or register to join the discussion
  • Overestimated

    It's just a very small outfit of freaks and nerds that actually care about specs. The majority of consumers doesn't even have the faintest clue about all the gobbledygook.

    Unfortunately technicians and marketing jerks spend way too much time on the internet listening to the tech nerds which lets them conclude that high numbers is what consumers are craving for.
    EnticingHavoc
    • Thankfully we're getting past the whole "speeds

      But "device X is Y% faster than device Z" is something that still comes up, and it's the less tech savvy people that will believe that line from the sales monkey at the carrier store when they're looking for their first smartphone.
      matthew_maurice
    • Specs.

      I am definitely a Geek, but I am not so confident in a spec that I let it overrun my logic. I almost choked when I saw that HTC only put a 4mp camera in the HTC One. The other top of the line smartphones seemed to sport 8mp cameras. But they tradeoff, and there are always tradeoffs when it comes to digital cameras, is that they have really good low light performance. Put the HTC One, the Galaxy S4, and the Nokia 1020 in low light, and then compare their photos, at 100%, and you pick the winner. On the other hand, this is not something I would want to trade resolution for as I do very little low light photography. Specs are really only important for the parts of the device that are most important for YOUR personal needs. If you don't use the camera in a smartphone, then does it matter if it has the worst camera in the field? People put a lot of store by graphics processors, while I couldn't care less about how fast they render games I will never play.
      rphunter1242
      • Re: compare their photos, at 100%

        No photographer does that, ever.

        Photographs are evaluated in their entirety, scaled to whatever medium the intended output is. In general, both the pixel quality (what you will see "at 100%") and resolution contribute. Good examples could be found in the DxO Mark measurement (www.dxomark.com).

        So in general, a "not so good at 100%" camera with higher resolution may produce better final image than "the best at 100%" camera with low resolution.
        danbi
      • GPUs

        Apple has changed the game with GPUs, and they are now used for everything from a math coprocessor to scrolling and animation.
        varase
  • Android is Linux

    Linux is the best
    Linux needs to fake benchmarks to appear to be the best
    Linux isn't really what it pretends to be
    Just for you SJVN, spin man, unravel it
    greywolf7
    • It's all about smartphone makers

      Not linux or android. Actually when stock android is used nothing is faked.

      And Nexus 4 do pretty well in non-graphic related benchmarks.
      AleMartin
      • That was some fast apologist-work

        Nice.
        LBiege
        • Just making things clear

          Mixing stuff it's not nice :)
          AleMartin
          • Now who would ever mix

            But you didn't clear anything up really.
            greywolf7
    • It's the hardware not the software

      Phone makers are pushing the limits of the hardware to fake results and make people want their phone. Nothing to do with the the OS.
      dsw0@...
  • Nothing new here

    Various manufacturers have been caught out trying to cheat at benchmarks across all sorts of different products. Who really believes car manufacturers MPG data ?
    Alan Smithie
    • Vehicle MPG data

      In Europe mpg data is determined using standardised, govt-fixed conditions.

      Thus whilst the numbers may or may not be a reflection of real-life useage (how do you define that anyway?), they are useful for comparisons.
      DAS01
    • Guess you missed the part where Apple, Motorola

      and the Google Nexus WEREN'T doing this.
      baggins_z
  • except..

    Didn't Apple do this in the past as well?

    Anyway, the Nexus line doesn't do it and the Nexus 5 supposedly wiped the floor with Apple's new 64bit 5S. :) We will have to wait and see though.

    As for the vendors cheating.. companies get greedy and cheat. Still not cool.

    I'm even happier that I switched to Nexus everything, except for the Asus Transformer series.
    ciph3ro
    • It seems Apple is not doing it

      But maybe in another life, at least they can be trusted now :)
      AleMartin
    • Not exactly. Back in the days when

      the PowerPC platform was finally being caught up to by Intel, Apple simply ignored everything except photoshop where the PowerPC still held a performance edge. A short time later, they dumped PowerPC and moved over to Intel.
      baggins_z
    • Re: Didn't Apple do this in the past as well?

      No.

      Why would they?
      danbi
    • Benchmark Cheating

      Nope, Apple doesn't do that.

      In fact, while everyone else (including Linux) was trying to optimize their kernel to produce high specmark scores, Apple was trying to optimize for low user interface latency. They're a company that produces high-end consumer devices, and they definitely know where their bread is buttered.

      Just take a look at the spec sheet on the iPhone 5s and you can plainly see they're not into specs - specs are for the guys in the crowd who are trying to differentiate themselves from the rest of the pack. Apple does what they do and really don't give a damn what everyone else is doing.

      Look at display size and NFC, what everyone else is saying their handset lacks. Apple could give a darn about NFC until there's a valid, universal use case for it. They'll go for iBeacon since the already have the radio for it.

      They'll probably go to a bigger display in the iPhone 6 if there are good enough displays available. And don't expect them to do 1080p - they'll probably do about 300-325 ppi and just drive the number of pixels the user can see.
      varase
  • EVERY vendor has done it.

    Anytime you try to run a benchmark, on any platform, the vendor tries to make theirs show up best.

    Ran across one trying to show system throughput - so what did they do? 1) overclock the system during reads/writes 2) hid the fact that they were writing to cache. Then reconfigure before actual use.

    Fast database throughput? repeat - only make the cache big enough for the entire database, and be sure nobody notices. Reconfigure before the reliability tests (power would be pulled from the system which would prevent the cache from getting flushed to disk).
    jessepollard