Benchmark vendor comes down on Samsung, Sony phones for gaming tests

Benchmark vendor comes down on Samsung, Sony phones for gaming tests

Summary: Last year, a number of Samsung phones were found to be falsely inflating Primate Labs' popular Geekbench 3 benchmarking tests. The software maker recently reported the results of its benchmark boosting detection and how it will deal with the gaming of its system.

SHARE:
17

In a post to the Primate Labs blog, company founder John Poole laid out the issue of phone makers' recent gaming of its benchmarking tests and the results of its in-house investigation.

Benchmark vendor comes down on Samsung, Sony phones for gaming its tests

Last year, several tech sites discovered that Samsung was fudging its test results with speed-boosting code. The artificial results inflated a few tests by as much as 20 percent, Poole reported.

Check Out: Samsung caught fudging benchmarks (again)

Poole said the company added a "boost detector" into Geekbench 3, which embedded a special report into each test result uploaded to the Geekbench database. Here's the company's list of gamed phones:

Samsung Galaxy Note 10.1 (2014)

Samsung Galaxy Note 2

Samsung Galaxy Note 3

Samsung Galaxy S 3

Samsung Galaxy S 4

Sony Xperia Z

Sony Xperia Z Tablet

Sony Xperia Z Ultra

Sony Xperia Z1

Sony Xperia ZL

The speed boost was found in Android 4.3 only, he wrote. Unlike the asterisks found in the Major League Record Books, Primate Labs pulled the problem results.

In order to combat benchmark boosting we have decided to exclude results from these devices running Android 4.3 from the Android benchmark chart. This way the results on the chart reflect the true performance, not the boosted performance, of each device. We have also added a list of excluded devices to the chart. We will continue to monitor the detector reports, and we will update this list if we discover other devices or Android versions that apply a benchmark boost.

There is one bit of good news that our detector uncovered — Samsung removed the benchmark boost from their Android 4.4 update. We hope that Sony follows Samsung's lead and also removes their benchmark boost from their Android 4.4 update as well.

Dudes, what were you thinking?

Topics: Samsung, Android, Smartphones

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

17 comments
Log in or register to join the discussion
  • uninformed and biased reporting

    You make it sound like Samsung/Sony were falsifying benchmark results.
    What they did was not falsifying the results but simply detecting benchmark software being executed and then ramping the CPU and GPU to the max instead of relying on automatic performance scaling that may never hit the max based on OS variables.
    IT IS STILL BENCHMARKING ACTUAL CPU AND GPU PERFORMANCE.
    You can simply rename the apk and it won't detect it as benchmark software and it won't manually max the CPU/GPU. The results are still very high but may be anywhere from 5-20% lower depending on heat, battery voltage, thread priorities etc.
    But hey, uninformed Android bashing is trendy.
    warboat
    • It's wrong in our industry to do it.

      Simple as that. Has nothing to do with Android. Whoever does it is wrong.
      IAmMarty
      • Hey I agree it's wrong

        But I'm telling it like it is rather than this sort of uninformed reporting which regurgitated Poole wrongly accusing Samsung of "embedding reports" and falsifying results.
        The benchmark results are still ACTUAL performance.
        The CPU and GPU still has to do the work to get those results..
        There is no falsifying performance going on as accused.
        Setting the conditions to maximise performance for benchmarks gives actual performance results achievable if the user set those conditions themselves. Admittedly, some of those conditions are hard to achieve by the average user if they don't have the ability to tweak the device to the extent of benchmark mode. The real issue is sustainable performance and that's always the problem with peak performance benchmarking.
        This article is trying to smear Samsung and Sony by ignorantly accusing them of outright result fudging which is not what is going on.
        Lets get the facts straight and not just regurgitate biased bullsh17.
        warboat
        • You're wrong.

          What matters is this was a deliberate attempt to mislead users.

          It's like claiming that it isn't cheating in cards to look at the other player's hands in a game of poker, because you still have to play the hand out.

          And no, real users could not repeat the process, in any real situation.

          Real situations don't repeat that precisely.

          And not only can they not perform the tweaks, but they aren't in a position to write off a few hundred devices along the way, for each situation that they want to push to the absolute limit.

          What Samsung set out to do was to deceive. and by deceiving, sell more product. i.e. to gain money by deception.

          The article is absolutely write to expose this in the strongest possible terms.
          Henry 3 Dogg
          • attempt to mislead

            yes, it's misleading. It's not right and should not be done.
            no, it's not like peeking at cards.
            yes, real users CAN repeat the conditions or even tweak beyond Samsung/Sony benchmark mode. The easiest way of course is to rename your app as Antutu or Quadrant or Geekbench and you get exactly the settings of the benchmark mode.
            Benchmarks on mobile devices is hardly consistent to begin with. Even in benchmark mode, power management will still kick in to slow things down when temperatures hit thresholds. It is not overclocking the device, it is not a speed boost but rather a disabling of speed restrictors. It does take away the choice in running the benchmark in normal modes and that's the real issue. However, the workaround is simply to rename the APK before installing.
            The benchmarks are not running on Samsung's test devices, they have to run on users phones, so if the benchmark mode is dangerous and blowing up devices, it is blowing up customer devices.
            The article is wrong in trying to over represent the cheating that's going on.
            The article is trying to deceive and distort the truth - a bigger crime than benchmark mode itself.
            It also singles (doubles??) out Samsung and Sony only and no mention of other players doing performance tweaks for benchmarks.
            The real problem with all this is the hype surrounding benchmark software and especially people like Poole thinking his benchmark represents real world performance and wants to be held as a gold standard for churning out a performance number. If you buy into that, then you are a sucker for the benchmark industry.
            warboat
          • Your fanboy is showing.

            SimonUK
        • Defending this deception is ridiculous

          It doesn't matter what benchmarks you can get for your hardware if that is not what users will be getting. Why are you trying to defend these guys? Those 2 manufacturers got caught trying for trick consumers. That is what they were doing.

          It is fine that you have your personal take away from this story. But you are wrong in your attack on those who focus on the desire to deceive to mislead consumers. And this is NOT Android bashing. It has nothing to do with the merits of the OS. In fact, it isn't even a big deal in terms of the merits of individual devices. All it is is exposing the unethical practices of manufacturers.

          All the defensive fans of different OSes need to relax when they see criticism of anything related to their OS. Save your indignation. When you hear bad news about something related to your favorite OS, take a step back and breath. Enough with the partisan electronics and the 'can do no wrong' / 'can do no right' approaches. Polarization about OSes is, well, stupid.
          DaveJMo
    • False

      The point was that Sony and Samsung were disabling power management techniques for benchmarks only. Meaning that the results displayed were not indicative of what could ever be achieved by real world apps. Power consumption and heat dissipation are important considerations in the mobile space. Neither Google nor Apple manipulated benchmarks in their own branded devices. It also appears Samsung has stopped in 4.4. So stop defending what they did before.
      KPOM1
      • That point was never made in this article

        The point made was fudging results and that is not the real truth.
        Nowehere in this article has the point about bypassing adaptive performance management been made.
        Its the difference between tweaking the boost to get actual racing timeslips and simply photoshopping the timeslips. HUGE DIFFERENCE and this article got it wrong accusing the latter is happening.
        While it is still evil, let's get the facts straight and not accuse them of cheating beyond their means.
        warboat
        • Photoshopping?

          Where did that analogy come from? You are projecting way too much.
          DaveJMo
    • And it wasn't cheating...

      when I aced my senior year history final after I got a copy of the questions that would be on it, because I still had to study so I'd know the answers.

      Seriously, you yourself acknowledge that because of their deliberate actions, their devices benchmarks are coming in 5-20% higher than during real world scenarios, but you don't think this is deceitful, especially when those same results are being used to compare Samsung and Sony devices against those of other OEMs who DIDN'T use the same tactics?

      Sounds like you have a good bit of bias yourself.
      TroyMcClure
      • Yes, it is cheating

        Knowing in advance the question for your final test while the rest of your fellow students does not is cheating. While they have to do a all night studying for any question that my come out, you on the other hand, just needed to study a few hours to only have the information needed to ace the test. It will be great if the teacher changes the question. Then what will you do, when all your study was for nothing and you do not know nothing for the answer of the new questions. People like you make me sick. Don't know nothing of the subject except the few questions on the test.
        jazzy2945
      • This all began with Qualcomm marketing trying to outdo Samsung

        They pushed the Snapdragon 800 against Samsung's own Exynos. They priced their 4G radio IC high enough to make the SD800 with built in 4G radio economically preferable.
        If Qualcomm priced their 4G radio competitively, you would not see the SD800 being overclocked as much to keep up with the Joneses.
        Run Droidfish Chess for CPU benchmarking and you will see that the Exynos versions of Note 3 sustains much higher computational performance than the SD800 versions. The Exynos in the Note 3 is also HMP capable which the SD800 is not, but the HMP capabilities has been disabled by Samsung for some reason. Heterogenous Multi Processing is the ability to scale the processing of each core (all 8, Not just 4) independently. The SD800 cannot do that, but the Exynos Octa 5 can but Samsung crippled it.......for now.
        Samsung showed the same chip in a tradeshow running all 8 big-little cores independently scaled because the other chip makers at that tradeshow were showing off 8 core HMP (allwinner, mediatek etc).
        Qualcomm has also got a deal with Samsung to have exclusive features in the SD800 version like 4k video recording which the Exynos versions are also capable of achieving. Not sure what the backroom deals are, but the players are not playing their cards straight up.
        warboat
    • warboat logic

      Heh heh. Good one.

      There's nothing wrong with mass-murder. All those victims were going to die some day anyway.
      varase
    • If they used speed boosting code

      To inflate the scores artificially then YES they did in fact falsify benchmark results. Perhaps the uninformed part is your constant defense of all things Android and Google.
      athynz
      • not the same

        It's like saying you dial it up to 2 bar of maximum boost when you're drag racing but set it at 1.2bar max on the street normally. You have the ability to hit the 2bar mode for short bursts ANYTIME ANYWHERE for maximum performance when needed.
        Do you tell people about your 1.2bar drag times or the 2 bar timeslips?
        This article is trying to say that Samsung and Sony photoshops the timeslips to brag about it.
        warboat
        • You lie

          Where did they mention photoshop? Why are you repeating that nonsense?

          The article is saying that the phone disabled throttling features for benchmarks when those features are ALWAYS in place otherwise. Some making up you own version of what the article says so that you can argue against a straw man.

          The manufacturers are not being misrepresented in the article. You are just reading something into it and then blaming the author. The contents are pretty straightforward. It is like an employee who only works when his manager is in the office.
          DaveJMo