X
Tech

Benchmark showdown: High-end Android smartphones

If you think that all 1GHz Android smartphones are equally quick, think again. ZDNet Germany's Christoph H Hochstätter shows that the performance of CPUs, RAM, flash and GPUs varies widely.
Written by Andy Smith, Contributor

If you think that all 1GHz Android smartphones are equally quick, think again. ZDNet Germany's Christoph H Hochstätter shows that the performance of CPUs, RAM, flash and GPUs varies widely.

The basic specification of all high-end smartphones is pretty standard: a 1GHz ARM processor; a 3.5in. to 4in. display with a resolution of about 480 by 800 pixels; 3D graphics acceleration; and around 512MB of RAM.

Internal flash memory varies widely though, from 512MB up to 32GB in Apple's iPhone 4. Android devices from HTC currently have 512MB of internal flash, while Samsung goes up to 16GB. Expansion via external SD cards is generally available for Android phones.

Another distinguishing feature is the camera: some smartphones have a front-facing camera (for making video calls) as well as the usual rear-mounted unit. An LED flash may or may not be present. All this adds up to big differences in picture quality.

What's not evident from the smartphone manufacturers' specifications is the fact that there are significant performance differences. Fortunately, these are clearly measurable.

ZDNet Germany uses Aurora Softworks' Quadrant Professional test suite for Android smartphones. In addition, we use Rightware's Browsermark, a browser benchmark specifically designed for Javascript-enabled smartphones.

CPU differences: Cortex-A8 v Snapdragon Scorpion core The CPU benchmarks for 1GHz smartphones running Android 2.1/Eclair (blue bars in Figure 1) do not vary greatly, ranging from 609 points for the Google Nexus One to 749 for the Samsung Galaxy S.

All of the 1GHz smartphones use ARMv7 processors with almost identical features. However, the Motorola Droid X and Samsung Galaxy S use a standard Cortex-A8 core, while the HTC Desire and the almost identical Google Nexus One use Qualcomm's Snapdragon processor with the (Qualcomm-developed) Scorpion core.

465937-450-327.jpg

Figure 1: Under Android 2.1/Eclair (blue bars) the 1GHz smartphones deliver similar CPU performance. Under Android 2.2/FroYo (green bars) with the Dalvik JIT compiler, Snapdragon CPUs with the Scorpion core show a marked improvement. The two tests marked with an asterisk are from benchmark producer Aurora Softworks' database. All other tests were performed by ZDNet Germany.

Although the Cortex-A8 and Scorpion cores share many similarities, the Qualcomm design delivers better performance. Both cores support the NEON set of SIMD instructions for integer and single-precision floating-point arithmetic, and the VFP extension for double-precision arithmetic.

The Scorpion core, however, can process SIMD instructions in 128-bit-wide chunks, while the Cortex-A8 is restricted to 64 bits. Scorpion also has a deeper pipeline, which includes VFP commands, while VFP commands are not pipelined in the Cortex-A8.

These differences should be measurable. Under Android 2.1, with no JIT compiler, only differences with different ROM versions on the same model can be detected.

The situation is completely different with Android 2.2/FroYo (green bars in Figure 1). The HTC Desire with CyanogenMod firmware and Google Nexus One score over 5,000 in the CPU benchmark — a 600-700 percent performance boost. The Desire with the original HTC ROM scores 4,250 points, while the Samsung Galaxy S only delivers a 96 percent boost. There are no test results for the Motorola Droid X with Android 2.2. Meanwhile, the HTC Legend, with a 600MHz ARMv6-based Qualcomm MSM7227 processor, shows a CPU performance increase of 109 percent between Android 2.1 and 2.2 — similar to the Galaxy S.

Anyone who has upgraded their HTC Desire or Google Nexus One to Android 2.2/FroYo will know that their phone has become much faster. However, it doesn't feel like a 6- or 7-fold speed increase. So what's the explanation?

The reason is that the Quadrant Professional CPU benchmark is very arithmetic-heavy: six out of twelve constituent tests are pure basic arithmetic benchmarks. Five routines are more practical: checksum; ZIP compression; H.264 videodecoding; AAC audiodecoding; and XML parsing. The final test measures the speed of branch prediction. The pure arithmetic tests can make use of a JIT compiler, such as the Dalvik VM introduced in Android 2.2. However, in practice, smartphones are rarely used for number crunching.

It's clear that FroYo's JIT compiler currently only delivers significant performance gains for Snapdragon CPUs with the Scorpion core. This in turn explains why, so far, only a beta version of Android 2.2 is available for the Cortex-A8-based Samsung Galaxy S — the JIT compiler is the outstanding feature of FroYo. For the widespread Cortex-A8 cores, used in many high-end Android smartphones, the JIT compiler needs to be optimized. A Cortex-A8 core will still be slower than a Scorpion core at the same clock speed, but the Scorpion's advantage may not be as much 260 percent.

Main memory speed Another essential factor determining smartphone performance is the speed of the main memory. Manufacturers' data sheets sometimes provide no information on the RAM, a bad habit introduced by Apple. 'Memory capacity' in such cases refers to the flash, not the main memory.

465938-450-327.jpg

Figure 2: HTC, which also manufactures Google's Nexus One, uses much slower memory than its rivals Samsung and Motorola. However, faster memory consumes more battery power.

The graph in Figure 2 shows that Motorola and Samsung have opted for faster memory than HTC. The quantity and speed of memory has a big effect on power consumption, although the interaction isn't always straightforward. For example, the Nexus One has 512MB of main memory, and up to Android 2.1/Eclair, Google only activated 256MB of it to save power. To use the full 512MB, you had to install your own kernel.

Although the full 512MB is available under Android 2.2/FroYo, Nexus One's battery actually lasts much longer. This is largely due to the Dalvik VM's JIT compiler. In idle mode, 1GHz smartphones are clocked much lower, typically around 250MHz. The voltage is reduced and power is saved, the clock speed being restored if the CPU is called upon. Thanks to the JIT compiler, the work is done quicker — that is, the CPU spends less of its time running at full speed.

Flash memory The I/O test in Figure 3 not only measures access to the file system in flash memory, but also database requests under SQLite. CPU speed therefore has some influence on the benchmark results.

465939-450-327.jpg

Figure 3: The Motorola Droid X comes with 8GB of internal flash memory. Four 2GB flash chips are addressed simultaneously, resulting in a big lead over the single-chip competition.

The Motorola Droid X easily beats its competitors, and the reason is relatively straightforward. The Droid X has 8GB of internal memory, and Motorola uses four 2GB flash chips, which are addressed in parallel via a controller. The remaining smartphones tested use a single flash memory chip.

The Droid X uses a technology similar to that employed in solid-state drives (SSDs), although SSDs usually address six to ten flash chips simultaneously, and are therefore much faster than single-chip SD cards or USB sticks. However, Motorola's flash chip layout comes with a trade-off, as the required controller uses extra power.

3D acceleration In the graphs below, especially for 3D acceleration, the Samsung Galaxy S leaves the competition in its wake. It uses a PowerVR SGX-540 GPU from Imagination Technologies, which has four texture units, a texture size of 2,048 and can process eight light sources. The Motorola Droid X uses the PowerVR SGX-530, with the same texture unit, texture size and light sources specifications. The benchmark shows the markedly improved performance of the SGX-540 compared to the SGX-530.

The HTC Desire and Google Nexus One use an Adreno-200 GPU based on AMD's Z430, which has only two texture units. The benchmarks show that 3D performance drops off significantly — something that's visible in practice. When the 3D benchmark is running, the Galaxy S displays fluid movement with frame rates of 40 to 50 frames per second (fps); on the HTC Desire you get a distinctly ragged 8 to 15fps.

This is especially important considering the upcoming (Q4 2010) Android 3.0 update, code-named Gingerbread. Google wants to deck out its interface with multiple 3D effects, perhaps making third-party UIs such as HTC Sense redundant. The value of 3D acceleration for Android users can only increase.

The Adreno-200 GPU has its advantages, though: unlike the PowerVR SGX series, it supports the frame buffer. This means that some Flash videos can play in the browser using less processing power, and thus consuming less power.

465940-450-327.jpg

Figure 4: 2D performance varies little among the smartphones tested by ZDNet; the two lowest scores (asterisked) come from the Quadrant benchmark database.

465941-450-327.jpg

Figure 5: The 3D benchmarks show the advantages of the PowerVR SGX-540 GPU in the Samsung Galaxy S. This will be especially important for the upcoming Android 3.0 (Gingerbread) upgrade.

Browser and JavaScript performance We measured browser performance using Rightware's Browsermark, which is specifically designed for smartphones.

Browsermark is based on profiling data from real web sites, and is more 'real-world' than the purely synthetic Quadrant benchmarks. It uses numerous tests with JavaScript and CSS. Although the speed of font rendering is crucial in 'normal' web sites, this is not considered in Browsermark.

The Samsung Galaxy S comes out on top, although this is not due to the Dalvik JIT compiler and the Snapdragon CPU's Scorpion core — the WebKit browser is a native Android application that does not use the Dalvik VM. The HTC Desire and Google Nexus One are only just behind the Galaxy S.

465942-450-410.jpg

Figure 6: The Samsung Galaxy S wins the browsermark tests, with HTC's Desire and the Google Nexus One close behind.

Conclusions If you want an Android smartphone, you have several manufacturers and many models to choose from. Fans of Apple's iPhone have it easier: they're restricted to one supplier, it's well known that iPhone 4 is faster than the iPhone 3G; the reception problems with the iPhone 4 are also extensively documented.

If you're interested in outstanding 3D performance, particularly under the upcoming Android 3.0, you should consider a Samsung Galaxy S. However, Samsung has not yet announced whether an Android 3.0 upgrade for the Galaxy S will be released. Manufacturers naturally want to re-sell new hardware when a new OS version comes out: after all, with free updates the vendor only boosts customer satisfaction, not sales.

If future-proofing is your main concern, HTC hardware is the best choice. HTC has put its drivers into the Android Open Source Project (AOSP), where they can easily be found in the source tree. Other manufacturers are keeping their drivers under wraps.

With the HTC Desire and the Nexus One, 'ROM cooks' can easily create their own builds directly from the AOSP, with no dependence on the manufacturer's code. Even if HTC delivers no more updates for a certain model, it's likely that future Android versions will appear on sites such as XDA Developers or Modaco.

If overall smartphone performance is your priority, be aware that almost every Android device performs badly in some benchmark discipline. It's not possible to give a general recommendation for the 'fastest' Android phone. You'll need to determine which features are most important to you.

Remember also that hardware development is ongoing. The next generation of 45-nanometer ARM processors is in the wings — the current generation has a feature size of 65nm. This die-shrink will further reduce power consumption.

Samsung already has already announced a 1GHz dual-core processor — the 'Orion', which is based on ARM's Cortex-A9 architecture. Multi-core technology is on its way to the smartphone and tablet markets.

Qualcomm is also working on faster processors, with two new CPUs the — QSD8272 and QSD8672 — set to appear in the fourth quarter of 2010. These chips also have two Cortex-A9 cores and are clocked at 1.5GHz. The GPU is upgraded to the Adreno-220. The only differences between the two processors are in the UMTS frequencies supported: bands 1 (2100MHz), 2 (1900MHz) and 5 (850MHz) in the QSD8272; bands 1 (2100MHz), 4 (1700MHz) and 8 (900MHz) in the QSD8672. UMTS band 1 is used in Europe.

So if you're seeking the best-performing smartphone, you can be sure that faster and better devices are just over the horizon — just as in the desktop and notebook markets.

Editorial standards