Samsung caught fudging benchmarks (again)

Samsung caught fudging benchmarks (again)

Summary: Benchmarkgate continues as Samsung is caught giving inflated benchmark scores to Note 3 via popular benchmarking app.

SHARE:
TOPICS: Apple, Samsung
24

According to a post at Ars Technica, Samsung got caught with its hand in the cookie jar – again. It appears that the company is shamelessly falsifying benchmark scores of its Galaxy Note 3 to make it appear faster than it actually is. 

In July 2013 Samsung was caught boosting benchmarks on the international Galaxy S4AnandTech found that the international versions of the phone would run the phone's GPU at 533MHz for certain popular benchmarks while limiting actual games to a slightly lower 480MHz.

This time, while testing the new Galaxy Note 3, Ars noticed something "fishy going on" because Samsung's 2.3GHz Snapdragon 800 was blowing the doors off LG's 2.3GHz Snapdragon 800. "What makes one Snapdragon so different from the other?"

Shenanigans. That's what.

It turns out that Samsung is artificially boosting the Note 3's benchmark scores by 20 percent with a special, "high-power CPU mode" that kicks in when the device runs a large number of popular benchmarking apps, like the popular Geekbench app. 

Geekbench 3 running on the Nexus 4, which idles normally—only one core is active, and it's running at the lowest possible clock speed.
Photo: Ars Technica

According to Ars:

Normally, while the Note 3 is idling, three of the four cores shut off to conserve power; the remaining core drops down to a low-power 300MHz mode. However, if you load up just about any popular CPU benchmarking app, the Note 3 CPU locks into 2.3GHz mode, the fastest speed possible, and none of the cores ever shut off.

I can't wait to hear Samsung explain this one.

Topics: Apple, Samsung

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

24 comments
Log in or register to join the discussion
  • Power management!

    Isn't that what these ARM processors are suppose to do? Good that the processors activate or kick in just when they are needed, right? As if blaming the manufacturer of "controlling" their products to act in maximum mode just to show that they have the fastest cpu's...this is bs!. I think it has something to do with their modified OS. If the benchmarks can show that cpu's indeed reach their maximum speeds then, truly the manufacturers are using their hardware and software combination to good use. It would be a waste to pay more for a high-end device when you can not even use these to their full potential, right?
    barzee
    • Read the article..

      My understanding after reading the article relates to the Samsung Note 3 only engaging "overdrive" when the device is running benchmark tests, not in normal day to day activities.

      People use benchmarks as a measure of normal functionality of a device, not "optimised" functionality designed to throw off benchmark results. Say you bought a device that appears on paper to outperform competition but still have a good battery life. If it turns out that one of your reasons for buying it was inaccurate, because the manufacturer "tweaked" the settings you would be understandably annoyed.

      In a nutshell, it boils down to the Brand. If you are perceived to cheat, you loose customers - period.

      Samsung doesn't need to use this type of tactic.
      MovingOn.Pmb
      • They actually need to use such tricks

        That's how they beat competition.
        dcdavy
      • Some companies are just bent

        Like Nokia getting caught falsifying the video passed off as taken with their smartphone, but really taken with a large dedicated video camera.
        Henry 3 Dogg
  • Why stoop to such behavior ?

    Much reminds me of the nonsense we saw in the realm of graphics cards not that long ago. A little cheating here and little tampering with benchmark files there and voila all of a sudden staggering numbers trickled in.

    That provides quite some evidence about the state of mind in Samsungs marketing/tech labs. Pimply teens drooling over specs .... sigh.
    EnticingHavoc
    • This is the second scandal involving the Samsung Note 3

      I also wondered why Samsung needs to behave so badly.

      The other scandal involving the Note 3 is the region-blocking of SIM cards, so if you travel to a different country you can't buy a Samsung phone and take it back home, as Samsung will block the SIM.

      Both that and the faked benchmarks does nothing to enhance Samsung's reputation.
      Vbitrate
      • Almost every oem does this..

        not only samsung
        htc, lg, asus are also in the business....

        http://www.phonearena.com/news/The-numbers-always-lie-HTC-LG-and-Asus-all-found-to-boost-benchmark-scores_id47914


        http://www.anandtech.com/show/7384/state-of-cheating-in-android-benchmarks
        smartshuaib
    • The Korians are an odd bunch

      While they make very high quality products and have great pride in the quality of their work they have a bad habit of cheating on specs as showen in this article. Same thing is going on with their cars, Hyundai has been nailed for over rating MPG on their cars and I have yet to see a review of a Hyundai that comes close to the advertised MPG.
      KBabcock75
      • Hardly an exclusively "Korian" thing

        Until the government put a stop to it, US automakers and all audio equipment makers lied shamelessly in their specs .

        Horsepower ratings were measured with no water pump, fuel pump or other power-consuming accessories attached.

        Stereo power output was specified in "Peak Music Power" (which measured the highest instantaneous peak power the amp could put out) at unspecified (and generally horrible) distortion levels - meaning that an amp rated by today's standards as maybe 12 watts could be advertised as 75 watts ... and then they tacked on "+/- 1 db", which let them overrate it 25% beyond THAT.
        fairportfan
        • True but...

          These were loop holes that, how ever dodgy, were never spelled out. Now that they are these companies are playing by the rules. The Korans know the rules and are cheating them and the real head scratchier is they make a good enough product that they don't have to.
          KBabcock75
  • Benchmark is not fake

    The benchmark is not fake, it shows de full power of the processor (4 cores). Samsung only put 3 cores on normal operations to manage de power consume. Probably an expert user can use the four cores, they are there!
    The benchmark reflect the real hardware inside de note 3. If you see the tests note 3 win against de LG without the full power.
    If you want to benchmark anything, you need to benchmark-it at the full power not with the power management that O.S. have.
    João Fernandes
    • An interesting spin but totally wrong interpretation on this topic

      Benchmark tests are designed to offer a set of tasks designed to evaluate multiple systems under the SAME controlled conditions. Thus, benchmark tests can evaluate the performance of one or more system characteristics and compare those results against the same results from different but similar devices.

      Your example describes a situation where one device is tested under a set of operating conditions that are DISSIMILAR from a set of tests performed on another device. Thus violating the definition of a "Benchmark" test.

      You should know that (and I suspect you do). I applaud your creative but disingenuous attempt to explain why Samsung chose to test their electronic device in the manner in which they did.

      But you didn't explain why Samsung published results for a particular benchmark program where, in fact, that was not the case. Samsung ran a benchmark test on their device but altered the parameters of that test thereby changing the test itself. In effect, Samsung didn't perform the same benchmark test but used the benchmark test's name in reporting their results.

      That's a contemptible and morally indefensible business practice, IMO.
      kenosha77a
      • Not so wrong...

        "Normally, while the Note 3 is idling, three of the four cores shut off to conserve power; the remaining core drops down to a low-power 300MHz mode. However, if you load up just about any popular CPU benchmarking app, the Note 3 CPU locks into 2.3GHz mode, the fastest speed possible, and none of the cores ever shut off."

        Reading the above text, Samsung didn't do any "overcloking", they only put the processor running at the fastest speed possible, turning off the power management settings (turn on all the cores of the processor).
        And do that only via software, if I was a developer, probably i can give the same instructions in my program to have de note 3 at full power.
        They didn't fake the results, the note 3 processor is capable of giving 2.3GHz at full power cores. That speed is real!
        That speed is possible with the power management on? NO
        A note 3 user, without knowledge to turn de power management off, can operate at 2.3GHz (with four cores on)? NO
        A "normal" note 3 user can notice the fulll power of the processor? probably when see the battery drain to fast....
        Can i drive my car to the limits? NO, but a professional pilot can...
        João Fernandes
        • I understand your point. However Samsung lied about the test results

          Again, for clarities sake, I will use an example from the Ars Technica article Jason's blog is based upon.

          http://arstechnica.com/gadgets/2013/10/galaxy-note-3s-benchmarking-adjustments-inflate-scores-by-up-to-20/

          Samsung reported performance numbers for their device after a particular benchmark test was performed. Let's call that test "Geekbench3.

          Ars Technica's Ron Amadeo recompiled the Geekbench3 test but called the test "Stealthbench" instead.

          He then ran the test and found a 20 percent reduction in the numbers (roughly) that Samsung cited.

          What Samsung did was program the device to key on the app word "Geekbench" and alter how the device's ARM processor ran that app.

          I must admit, it's a criminally creative way to fool customers into thinking they have a device that performs much better than a competitors product.

          Samsung cited a "Geekbench" result without informing the prospective reader of those results that they had modified the Android operating system to allocate ALL system resources to that app while it was running. Again, something that would normally not happen.

          That is a lie of omission. The BEST OF LIES.

          Now, you are correct in all your assertions that programmers COULD hardwire their app code to overwrite the device's Android OS normal programming in order to give their app ALL the devices system resources.

          They "could" do that.

          But they would be financial fools to do so since consumers would understand that this particular app was hogging all the system resources and draining the device's battery charge. Those same consumers would also notice that other apps running simultaneously would show an adverse performance hit whenever this other "system resource hogging app" was running.

          Word-of-mouth would get around and other consumers would start to shy away from buying or installing this particular app.
          kenosha77a
  • Why is this article in Apple section

    ?
    Nova_Thor
    • Nova_Thor

      I have no idea why.

      Sumsung still is one of the most unethical companies out there. BOYCOTT SAMSUNG!!
      TimeForAChangeToBetter
      • Samesong deserved to be called out in any part of the ZDNetiverse.

        Even the Apple section.
        Champ_Kind
  • Bozo eradicator

    $am$ung is really shameless criminal.
    heispark
    • Samsung == copy machine

      A very efficient one indeed
      dcdavy
  • Fudge this

    Another author who feels there is real truth in marketing. LOL
    sickntired44