Benchmark showdown: High-end Android smartphones

Benchmark showdown: High-end Android smartphones

Summary: If you think that all 1GHz Android smartphones are equally quick, think again. ZDNet Germany's Christoph H Hochstätter shows that the performance of CPUs, RAM, flash and GPUs varies widely.


If you think that all 1GHz Android smartphones are equally quick, think again. ZDNet Germany's Christoph H Hochstätter shows that the performance of CPUs, RAM, flash and GPUs varies widely.

The basic specification of all high-end smartphones is pretty standard: a 1GHz ARM processor; a 3.5in. to 4in. display with a resolution of about 480 by 800 pixels; 3D graphics acceleration; and around 512MB of RAM.

Internal flash memory varies widely though, from 512MB up to 32GB in Apple's iPhone 4. Android devices from HTC currently have 512MB of internal flash, while Samsung goes up to 16GB. Expansion via external SD cards is generally available for Android phones.

Another distinguishing feature is the camera: some smartphones have a front-facing camera (for making video calls) as well as the usual rear-mounted unit. An LED flash may or may not be present. All this adds up to big differences in picture quality.

What's not evident from the smartphone manufacturers' specifications is the fact that there are significant performance differences. Fortunately, these are clearly measurable.

ZDNet Germany uses Aurora Softworks' Quadrant Professional test suite for Android smartphones. In addition, we use Rightware's Browsermark, a browser benchmark specifically designed for Javascript-enabled smartphones.

CPU differences: Cortex-A8 v Snapdragon Scorpion core The CPU benchmarks for 1GHz smartphones running Android 2.1/Eclair (blue bars in Figure 1) do not vary greatly, ranging from 609 points for the Google Nexus One to 749 for the Samsung Galaxy S.

All of the 1GHz smartphones use ARMv7 processors with almost identical features. However, the Motorola Droid X and Samsung Galaxy S use a standard Cortex-A8 core, while the HTC Desire and the almost identical Google Nexus One use Qualcomm's Snapdragon processor with the (Qualcomm-developed) Scorpion core.

Figure 1: Under Android 2.1/Eclair (blue bars) the 1GHz smartphones deliver similar CPU performance. Under Android 2.2/FroYo (green bars) with the Dalvik JIT compiler, Snapdragon CPUs with the Scorpion core show a marked improvement. The two tests marked with an asterisk are from benchmark producer Aurora Softworks' database. All other tests were performed by ZDNet Germany.

Although the Cortex-A8 and Scorpion cores share many similarities, the Qualcomm design delivers better performance. Both cores support the NEON set of SIMD instructions for integer and single-precision floating-point arithmetic, and the VFP extension for double-precision arithmetic.

The Scorpion core, however, can process SIMD instructions in 128-bit-wide chunks, while the Cortex-A8 is restricted to 64 bits. Scorpion also has a deeper pipeline, which includes VFP commands, while VFP commands are not pipelined in the Cortex-A8.

These differences should be measurable. Under Android 2.1, with no JIT compiler, only differences with different ROM versions on the same model can be detected.

The situation is completely different with Android 2.2/FroYo (green bars in Figure 1). The HTC Desire with CyanogenMod firmware and Google Nexus One score over 5,000 in the CPU benchmark — a 600-700 percent performance boost. The Desire with the original HTC ROM scores 4,250 points, while the Samsung Galaxy S only delivers a 96 percent boost. There are no test results for the Motorola Droid X with Android 2.2. Meanwhile, the HTC Legend, with a 600MHz ARMv6-based Qualcomm MSM7227 processor, shows a CPU performance increase of 109 percent between Android 2.1 and 2.2 — similar to the Galaxy S.

Anyone who has upgraded their HTC Desire or Google Nexus One to Android 2.2/FroYo will know that their phone has become much faster. However, it doesn't feel like a 6- or 7-fold speed increase. So what's the explanation?

The reason is that the Quadrant Professional CPU benchmark is very arithmetic-heavy: six out of twelve constituent tests are pure basic arithmetic benchmarks. Five routines are more practical: checksum; ZIP compression; H.264 videodecoding; AAC audiodecoding; and XML parsing. The final test measures the speed of branch prediction. The pure arithmetic tests can make use of a JIT compiler, such as the Dalvik VM introduced in Android 2.2. However, in practice, smartphones are rarely used for number crunching.

It's clear that FroYo's JIT compiler currently only delivers significant performance gains for Snapdragon CPUs with the Scorpion core. This in turn explains why, so far, only a beta version of Android 2.2 is available for the Cortex-A8-based Samsung Galaxy S — the JIT compiler is the outstanding feature of FroYo. For the widespread Cortex-A8 cores, used in many high-end Android smartphones, the JIT compiler needs to be optimized. A Cortex-A8 core will still be slower than a Scorpion core at the same clock speed, but the Scorpion's advantage may not be as much 260 percent.

Main memory speed Another essential factor determining smartphone performance is the speed of the main memory. Manufacturers' data sheets sometimes provide no information on the RAM, a bad habit introduced by Apple. 'Memory capacity' in such cases refers to the flash, not the main memory.

Figure 2: HTC, which also manufactures Google's Nexus One, uses much slower memory than its rivals Samsung and Motorola. However, faster memory consumes more battery power.

The graph in Figure 2 shows that Motorola and Samsung have opted for faster memory than HTC. The quantity and speed of memory has a big effect on power consumption, although the interaction isn't always straightforward. For example, the Nexus One has 512MB of main memory, and up to Android 2.1/Eclair, Google only activated 256MB of it to save power. To use the full 512MB, you had to install your own kernel.

Although the full 512MB is available under Android 2.2/FroYo, Nexus One's battery actually lasts much longer. This is largely due to the Dalvik VM's JIT compiler. In idle mode, 1GHz smartphones are clocked much lower, typically around 250MHz. The voltage is reduced and power is saved, the clock speed being restored if the CPU is called upon. Thanks to the JIT compiler, the work is done quicker — that is, the CPU spends less of its time running at full speed.

Flash memory The I/O test in Figure 3 not only measures access to the file system in flash memory, but also database requests under SQLite. CPU speed therefore has some influence on the benchmark results.

Figure 3: The Motorola Droid X comes with 8GB of internal flash memory. Four 2GB flash chips are addressed simultaneously, resulting in a big lead over the single-chip competition.

The Motorola Droid X easily beats its competitors, and the reason is relatively straightforward. The Droid X has 8GB of internal memory, and Motorola uses four 2GB flash chips, which are addressed in parallel via a controller. The remaining smartphones tested use a single flash memory chip.

The Droid X uses a technology similar to that employed in solid-state drives (SSDs), although SSDs usually address six to ten flash chips simultaneously, and are therefore much faster than single-chip SD cards or USB sticks. However, Motorola's flash chip layout comes with a trade-off, as the required controller uses extra power.

3D acceleration In the graphs below, especially for 3D acceleration, the Samsung Galaxy S leaves the competition in its wake. It uses a PowerVR SGX-540 GPU from Imagination Technologies, which has four texture units, a texture size of 2,048 and can process eight light sources. The Motorola Droid X uses the PowerVR SGX-530, with the same texture unit, texture size and light sources specifications. The benchmark shows the markedly improved performance of the SGX-540 compared to the SGX-530.

The HTC Desire and Google Nexus One use an Adreno-200 GPU based on AMD's Z430, which has only two texture units. The benchmarks show that 3D performance drops off significantly — something that's visible in practice. When the 3D benchmark is running, the Galaxy S displays fluid movement with frame rates of 40 to 50 frames per second (fps); on the HTC Desire you get a distinctly ragged 8 to 15fps.

This is especially important considering the upcoming (Q4 2010) Android 3.0 update, code-named Gingerbread. Google wants to deck out its interface with multiple 3D effects, perhaps making third-party UIs such as HTC Sense redundant. The value of 3D acceleration for Android users can only increase.

The Adreno-200 GPU has its advantages, though: unlike the PowerVR SGX series, it supports the frame buffer. This means that some Flash videos can play in the browser using less processing power, and thus consuming less power.

Figure 4: 2D performance varies little among the smartphones tested by ZDNet; the two lowest scores (asterisked) come from the Quadrant benchmark database.

Figure 5: The 3D benchmarks show the advantages of the PowerVR SGX-540 GPU in the Samsung Galaxy S. This will be especially important for the upcoming Android 3.0 (Gingerbread) upgrade.

Browser and JavaScript performance We measured browser performance using Rightware's Browsermark, which is specifically designed for smartphones.

Browsermark is based on profiling data from real web sites, and is more 'real-world' than the purely synthetic Quadrant benchmarks. It uses numerous tests with JavaScript and CSS. Although the speed of font rendering is crucial in 'normal' web sites, this is not considered in Browsermark.

The Samsung Galaxy S comes out on top, although this is not due to the Dalvik JIT compiler and the Snapdragon CPU's Scorpion core — the WebKit browser is a native Android application that does not use the Dalvik VM. The HTC Desire and Google Nexus One are only just behind the Galaxy S.

Figure 6: The Samsung Galaxy S wins the browsermark tests, with HTC's Desire and the Google Nexus One close behind.

Conclusions If you want an Android smartphone, you have several manufacturers and many models to choose from. Fans of Apple's iPhone have it easier: they're restricted to one supplier, it's well known that iPhone 4 is faster than the iPhone 3G; the reception problems with the iPhone 4 are also extensively documented.

If you're interested in outstanding 3D performance, particularly under the upcoming Android 3.0, you should consider a Samsung Galaxy S. However, Samsung has not yet announced whether an Android 3.0 upgrade for the Galaxy S will be released. Manufacturers naturally want to re-sell new hardware when a new OS version comes out: after all, with free updates the vendor only boosts customer satisfaction, not sales.

If future-proofing is your main concern, HTC hardware is the best choice. HTC has put its drivers into the Android Open Source Project (AOSP), where they can easily be found in the source tree. Other manufacturers are keeping their drivers under wraps.

With the HTC Desire and the Nexus One, 'ROM cooks' can easily create their own builds directly from the AOSP, with no dependence on the manufacturer's code. Even if HTC delivers no more updates for a certain model, it's likely that future Android versions will appear on sites such as XDA Developers or Modaco.

If overall smartphone performance is your priority, be aware that almost every Android device performs badly in some benchmark discipline. It's not possible to give a general recommendation for the 'fastest' Android phone. You'll need to determine which features are most important to you.

Remember also that hardware development is ongoing. The next generation of 45-nanometer ARM processors is in the wings — the current generation has a feature size of 65nm. This die-shrink will further reduce power consumption.

Samsung already has already announced a 1GHz dual-core processor — the 'Orion', which is based on ARM's Cortex-A9 architecture. Multi-core technology is on its way to the smartphone and tablet markets.

Qualcomm is also working on faster processors, with two new CPUs the — QSD8272 and QSD8672 — set to appear in the fourth quarter of 2010. These chips also have two Cortex-A9 cores and are clocked at 1.5GHz. The GPU is upgraded to the Adreno-220. The only differences between the two processors are in the UMTS frequencies supported: bands 1 (2100MHz), 2 (1900MHz) and 5 (850MHz) in the QSD8272; bands 1 (2100MHz), 4 (1700MHz) and 8 (900MHz) in the QSD8672. UMTS band 1 is used in Europe.

So if you're seeking the best-performing smartphone, you can be sure that faster and better devices are just over the horizon — just as in the desktop and notebook markets.

Topics: Mobility, Hardware, Smartphones

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • Is this really a good thing?

    I'm not a big fan of Android, but I still see it as a major step up from what was the norm for smartphones before the iPhone. I see the advantages over the iPhone in the perceived "freedom of choice" that it has.

    However, with the different OS revisions that will always be in the wild, and the different hardware, is this really a good thing? I'm asking a serious question here.

    I see the iPhone as successful because it is more of an appliance like device, like a console. Other then storage size (which has no affect on actual performance), all "current" iPhones are the same.

    It's not necessarily the amount of power that your hardware has, its about what the developers are forced to deal with. Give a developer a single device with specific specification and they will (assuming they are good) build the best product they can for that device, or at least try to. This has worked well for consoles. For the most part, an Xbox 360 is an Xbox 360... the PS3 is a PS3, and the Wii is a Wii. There might be some changes as revisions increase but for the most part they are the same. When you buy a game for a PS3 you don't have to see if you have a PS3 or a PS3 Galaxy Incredible.

    Now, if developers build apps for the lowest end device, then everyone can be assured the app will run well. Of course, then why have a more powerful device? If apps will only run well on a faster cpu or better memory, then how do you deal with those that have a slightly lower end model?

    Of course this really only matters in real world use. If all the devices have different components, and the benchmarks tell you that one does X points better then the other, it only really matters if that translates to real world use (which benchmarks don't always do).

    These are just some thoughts. None of this may matter, and the Android market may just automatically deal with it. Everyone I know with an Android phone pretty much treats it like any other phone with great web browsing capabilities and a couple games. So the difference in specs might turn out to really be a non-issue. (In which case, does it just add to confusion as to which device to get when it doesn't really matter? I don't know).
    • RE: Benchmark showdown: High-end Android smartphones


      I think there are two issues here.

      First, most iPhone users are able to upgrade to the latest OS. That's something that Apple has done really well and kudos to them. This is something Google simply has to address. But in spite of this, there are still fragmentation with iOS. The original iPhone and iPhone 3G can't get multi-tasking. 3GS doesn't get HDR. iPads don't have 4.0 and iPhone apps have to be coded for the larger size (not by just scaling, but really utilizing the extra space). The problems on Android are no different. And neither really is a huge problem for good developers. They should be easily able to handle multiple screen resolutions, hardware and software capabilities by coding to the APIs and gracefully degrading when hardware/software features are not available.

      The second issue is of having different hardware at the same OS version - which is what this article is about. IMO, this is a huge advantage for the Android. Manufacturers are going to differentiate themselves using hardware and consumers win because they get to pick the phone that delivers most of the features they are interested in. eg. the Samsung Galaxy S with its awesome display, DLNA and 3D performance will appeal to people leaning more towards a media phone. The T-Mobile G2 will appeal to someone who prefers a keyboard and a stock UI. I think these are wonderful choices and this is where Android completely outshines the iPhone. Remember that these are all phones running the same OS and so have access to pretty much every app in the Market for that version.
      • Not to mention...

        Replaceable battery. Upgradeable flash memory.
      • And . . .


        on Android you get integrated Google spyware. Some fools think that is an advantage.
      • RE: Benchmark showdown: High-end Android smartphones


        with iOS4 you get the odd integrated fanatical zealots complete with tinfoil hats, totally dismissive of even good competition, deceiving themselves into thinking that mindset is an advantage.

        Dude; don't spoil it for all the non-fanatical iOS4 folk out there.
    • RE: Benchmark showdown: High-end Android smartphones

      Multiple OS in the wild....this is not an issue. Their paradigm is no different than PCs. There is XP, Vista, win 7 in the wild. Its not an issue. There are multiple PC vendors, it is also not a problem.

      At this moment in time, there is a huge difference between android 1.5 and 2.2 but thats because the OS is still relatively immature. And since people buy new phones more frequently than computers, they will catch up easily. Plus, its much easier to upgrade mobile OS. So this time next year, this won't even be a significant issue.
      • Plus their are a lot of options

        in the market. I have a mid range Android (MyTouch Slide). As of yet I haven't found a single program that won't run on my phone. I have read in review columns a few that don't (usually the Evo). However, there is almost always another program that does the same thing that WILL run on the other platform. So far I've been able to find something to do what I want for my phone and to date I haven't opened up my wallet once to do so (I try to pay attention to ads and click on them from time to time thus supporting the developers of said free stuff).
        I read something every day talking about how "fragmented" the Android system is. I say a better word would be consumer friendly. Android provides to consumers what Microsoft has - an OS that works with a lot of hardware options. The next step is for Google to come up with some way for the user base to self-upgrade with some kind of installer kit that has a wide range of hardware drivers for the various devices, thereby bypassing the ridiculous manufacturers waiting period for these updates.
    • RE: Benchmark showdown: High-end Android smartphones

      @tk_77 You are a little off with consoles. The Xbox 360 has multiple levels of processor speeds that you can buy. Granted you don't have to worry about which one you have to see if you can play a game. The PS3 may not have seperate levels but earlier versions were backwards compatible with PS2 games whereas newer versions do not. Plus you can load a Linux OS on the PS3 in order to play games you wouldn't normally be able to. Off the topic of the article I know but wanted to clarify.
    • RE: Benchmark showdown: High-end Android smartphones

      @tk_77<br><br>Also consider that if every Android phone had the same performance, that would necessarily mean that every Android phone would use the exact same components.<br><br>Apart from reducing competition amongst manufacturers, it would make the platform more vulnerable to component shortages. For instance, when Samsung suffered from AMOLED shortage, HTC and Motorola who bought these displays could switch over to LCD screens. If the platform required AMOLED screens for the sake of uniformity, it would have hurt sales.
  • see as single page?

    Please? Most websites have that feature, to "print" as a single page or view as a single page. I HATE having to click through SEVEN two-paragragh pages.
    • RE: Benchmark showdown: High-end Android smartphones

      Just hit 'Print'. Please use some creativity instead of HATE. :)
    • RE: Benchmark showdown: High-end Android smartphones

      @ChazzMatt The reason for spreading an article over seven pages is to increase the number of page hits and the number of ads served. Just be happy it is not fourteen one-paragraph pages.
  • Where does the Evo fit in this?

    maybe I missed it in the article, but where does the Evo fit in to this comparison?
    • RE: Benchmark showdown: High-end Android smartphones

      @tsrich That was my first question as well. I just read an article on ZDNet this week where the author listed it as the number 1 Android phone. How could it be left off of this list? It looks like a European study and maybe the EVO is not readily available there?
  • RE: Benchmark showdown: High-end Android smartphones

    Just curious... Why did you ignore SonyEricsson Xperia X10?
    It's equipped with Snapdragon 1GHz processor, has WVGA display and so on. Do you think that Xperia isn't High-end Android smartphone?
    (I know that it still works under Android 1.6. from the Dinosaurs era. But is it really a deadly sin?)
    Serge Sereda
    • RE: Benchmark showdown: High-end Android smartphones

      @Serge Sereda Well, X10 has not been mentioned in any of the android related reviews I have read. Thanks to the aftersales support of Sony Ericsson. Hoping to get the old 2.1 on my X10 this month. And people are already talking about getting 3.0 on their mobiles. :(
    • RE: Benchmark showdown: High-end Android smartphones

      @Serge Sereda you have to remember that the study was done in Germany. The device may not be available there (see previous comment about the EVO)
      • RE: Benchmark showdown: High-end Android smartphones

        @russdwright@... I've specially checked - SE Xperia X10 was available in Germany since April 2010 - Thus, the real reason is in something else...
        Serge Sereda
  • RE: Benchmark showdown: High-end Android smartphones

    This article was written prematurely and is pretty meaningless because it compares a custom rom to Froyo to Eclair and then tries to judge the devices based on the results. Sorry but until all these devices carry Froyo, the results have to be taken with a truckload of salt.<br><br>This article speaks more about Froyo (optimized for the snapdragon)Vs Eclair than the actual differences in hardware performance.<br><br>Only when everyone's on a level playing field with regards to the OS will I take your results seriously.
  • RE: Benchmark showdown: High-end Android smartphones

    Where's the Evo in all of this?