Flurry report on iOS vs. Android allegiance called into question

Flurry report on iOS vs. Android allegiance called into question

Summary: A widely circulated report by Flurry Analytics uses dubious data to draw unsupportable conclusions.

SHARE:

Last week Flurry Analytics released a report on iOS vs. Android titled "App Developers Signal Apple Allegiance Ahead of WWDC and Google I/O". The report has been widely circulated around the internet, and generally accepted as factual with little or no critical review. A ZDNet investigation shows that the report's math is flawed and its charts are misleading, thus throwing doubt onto all of its conclusions.

Flurry's report is broken up into four sections, and we found problems with all four. Two sections in particular, though, show the most glaring issues: one on new project starts, and one on platform fragmentation.

Section 1: "Oh Captain, My Captain"

The Flurry report starts with a section on new project starts. Here's a chart from Flurry's report that shows the number of new projects started on both platforms:

Using this data, the report draws the conclusion that "Apple continues to garner more support from developers.  For every 10 apps that developers build, roughly 7 are for iOS".

The problem is that the data only tracks projects that use Flurry's SDK.

The Flurry SDK is a library that is linked into your application that sends data back to Flurry about how many people start your app, how often they use it, and for how long. Suppose that on Android, many developers use an alternative library such as Google Analytics. This would make a big difference in how the numbers are interpreted, wouldn't it?

You can see this is the case according to analysis by AppBrain which shows Flurry is used by only 6.58% of app apps in the Android Market:

If you'll excuse some quick, back-of-the-envelope calculations, Google claims 500,000 apps in their market, which works out to about 35,000 apps using Flurry. According to Flurry, their SDK is used by 185,000 apps, which comes to 150,000 for iOS and 35,000 for Android. Apple claims about 600,000 apps in the Apple app store, which means roughly 25% of iOS apps use Flurry.

Now, in Q1 2012, the Flurry chart shows approximately 13,000 Flurry new project starts on iOS, and 6,000 new starts on Android. That's how they got their '7 out of 10 are iOS' figure, because 13,000/(13,000 + 6,000)=0.72. But if you take into account the percentage of apps on each platform that use Flurry, you get 52,000 new projects on iOS total (13,000/0.25), vs. 91,000 new projects on Android total (6,000/0.0658).

This leads to quite a different conclusion: namely that there are nearly twice as many projects being started each quarter on Android than on iOS.

Next we'll consider Android's "F" word: Fragmentation.

Section 3: "Android Fragmentation Pain"

Another attention grabbing chart in the Flurry report is one that deals with how many different kinds of Android devices there are. Here it is:

Looks bad, doesn't it? Well, take a closer look. The chart lists each device individually, so for example it has 12 different slices for Samsung. Many of these are the same device running the same OS version. Here's how the chart would look if you combined all the Samsung pieces:

While compiling this, something odd stood out. Look at the black arrows. Flurry split one model, the Samsung Galaxy S II, into three slices which further magnifies the appearance of fragmentation. Doesn't anybody read these things?

The report argues that "each additional device a developer supports will deliver only a small increase in distribution coverage". In fact, Android developers do not support each device individually. Even the biggest developers have found that they only need about 10 to 15 devices in their test labs to cover nearly all the market.

Given this, we don’t see how the data supports Flurry's conclusion that "Android fragmentation appears to be increasing, driving up complexity and cost for developers". There are some costs, surely, but it seems that Flurry is using misleading charts to greatly exaggerate the issue.

The Flurry report has other sections and conclusions, but given the flaws in the two we have examined here, the findings of the entire report should be called into question. Flurry was contacted twice and given an opportunity to disagree with the findings in this story, but their only comment was "We don't have any comment".

Topics: Android, Smartphones, Security, Mobility, Mobile OS, Hardware, Google, Apps, Apple, Software Development

Ed Burnette

About Ed Burnette

Ed Burnette is a software industry veteran with more than 25 years of experience as a programmer, author, and speaker. He has written numerous technical articles and books, most recently "Hello, Android: Introducing Google's Mobile Development Platform" from the Pragmatic Programmers.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

27 comments
Log in or register to join the discussion
  • Won't be trusted by developers then!

    Obviously Flurry is possibly best avoided by Android developers if they do pull negative stunts like this.
    albionstreet
  • Both

    Both the Flurry article and your are misleading.

    There are lots of apps that don't use any of these analytics spy kits (like for example AutoCAD WS) so both parties claims are based on rumours not facts.
    danbi
    • Shows Proof in Inaccuracy

      Dont think Ed was looking to republish the data, just used his numbers to show it is very likely flawed. But yes, including all other potential scenarios the numbers are even worse.

      So yes, you are right it is even worse.
      So no, your wrong in the authors intent.
      rhonin
  • Apple is still better.

    Get it through your thick skull. No matter WHAT Google does, it just can't beat the iOS juggernaut. Apple perfected, honed, and still keeps making better and better the user experience on smartphones, so no matter WHAT proof you come up with saying this and that about Android vs. iOS, you'll always be in a losing battle.

    It's o.k. to be second fiddle. It's just - o.k....
    MrSmartyPants
    • lol!

      Better? Give details on how Apple's locked-down, bland, limited functionality, one-way-fits-all OS is "better" than Google's...and don't give the "its more polished" or "its simpler to use" nonsense.
      trob6969
    • Just LOVE this comment in an argument about evidence....

      ...no matter WHAT proof you come up with saying this and that about Android vs. iOS...

      Well said. The evidence doesn't matter because my iphone is shiny. La la la la la.... not listening.....
      billyboy1972
  • I'm shocked, shocked to find questionable mobile market research!

    "Your winnings, sir."
    matthew_maurice
  • Another Note on the F word...

    If Apple pushes iOS 5 to the 3GS but it isn't as feature rich as the iPhone 4 version of iOS 5 and again isn't as feature rich as iOS 5 on the iPhone 4s... Is this not the same situation as Android is in with 2.3.x and 4.x? Crud, Apple's Fragmentation might be worse because three versions of iOS 5 could be on 1/3 of all iDevices running iOS 5... Of course this gets worse when the 3GS never gets iOS 6 but it is still for sale at AT&T.
    slickjim
    • Your argument is backwards.

      Any given version of iOS always has the same features, regardless of whether a device that runs it can take advantage of them all. Are you saying that Windows 7 Pro on my notebook with a single drive "isn't as feature rich" as Windows 7 Pro on my server with a software RAID?

      And as for "[i]Of course this gets worse when the 3GS never gets iOS 6 but it is still for sale at AT&T.[/i]" When has Apple ever sold a device that couldn't run the shipping OS? Beyond that, when has Apple ever sold more than 3 models of iPhone. When the next "new iPhone" ships with iOS 6, expect the 3GS to go away, and you can be certain that iOS 6 will run on iPhone 4 and 4S.
      matthew_maurice
      • I think you're kind of missing a point here.

        Fragmentation isn't simply 'does the OS support something'. If that's all it is - then the Android problem is simple: target Android 2.2 and you're done. That would cover nearly 100% of all devices out there.

        It's more complex than that. Writing to a feature in iOS that isn't supported in a given model is still fragmentation. It means now I have to decide if some feature that I want to include should be included. Say I want to add barcode scanning... if all iPads had a camera, then I could just write it one way. But they don't, so I have to deal with iPads with and without. Not to mention having to actually have two versions of the app, one for each size of screen. And then there's focus - the iPhone's camera didn't get good enough for barcode reading until the iPhone 4. Do I support it for the 3s?

        In reality, it's just not that hard to write for Android. If you don't use a device specific feature - like NFC - then there's little to do. Heck, even screen sizes aren't a huge issue - the 'autolayout' feature in iOS6 has been in Android almost from the beginning. It's only developers who think like iOS programmers who end up having problems. Windows developers have dealt with the whole screen resolution issue for decades - so have MacOS programmers.

        As for Apple not selling a device that doesn't support the latest OS, that's wrong headed in two directions. First, Apple isn't the main seller of devices - the cellcos are. And they sell their stock until it's gone. So yes, it's quite possible for a device to be sold while no longer updated.

        But when it comes to phones, it's not just 'what's there when it sold'. Most phones are sold on contracts that run two to three years - and you can't just arbitrarily upgrade as that costs a LOT of money. So if Apple has a model that's shipping and is EOLed before all the contracts are up (say 3 years after the last one is 'officially' old) then it's the same problem.

        Which is what happened to me. I bought an iPhone 3G on a three year contract and it was EOLed by Apple about a year and a half into the contract. (It also died two weeks after the warranty expired, forcing me to by a new one - and because of how Apple works, I couldn't upgrade then either - I could only buy the same model if I wanted the 'refurb' price).

        I think in reality it's not that there's no fragmentation with iOS - just that the iOS developers tend to ignore it. And the fragmentation issues with Android are way overblown by iOS programmers who a little to lazy and spoiled.
        TheWerewolf
      • I never said anything about fragmentation.

        @TheWerewolf I simply said that his argument is a [i]non sequitur[/i]. Fragmentation does exist, and you list some good examples of how it effects iOS, but unless the devices never changed you're going to see the problems you indicate. However all your examples also apply to Android where they are orders of magnitude worse. In reply I'd say "yeah, and if a developer is going to face fragmentation, doesn't it make sense to go with the platform that has the least of it?"
        matthew_maurice
    • Peter get over it.

      You are expecting to run a 3 generations younger (ios 6) version to run a 4 generations older hardware (iphone 3gs), you are simply naive here. Also I don't see Apple pushing iPhone 3GS if it can't run iOS 6. They never did that with Original iPhone or iPhone 3G, so they wont.
      Ram U
  • Thank you ! ...

    for providing some critical analysis. I very rarely put 100% faith in any article, study, etc.., but it is nice to have Tech writers step in periodically and add value. Makes up for the eye candy (i.e. photo Galleries that require 10 clicks to see something that could be shown on 1 or 2 web pages).
    Now back to the subject at hand, I really don't care if there are 9 times as many games on iOS as Android (or Windows Phone). I really just care if most of the apps that I want are available on the platform I chose. In that case, I think Android and iOS (and possibly Windows Phone) aren't doing too bad. That is really what it boils down to.
    jkohut
    • Bummer Ed used really bad assumptions and data

      And came out with way wrong numbers.
      Bruizer
  • apple answered this one

    Lets see Android makers feel they can compete based on their products where apple feels it needs to use the courts to block android phones. Kind of says where things are going doesn't it?
    kroguej@...
    • Dreams don't make reality

      Just because that's your fantasy view of things does not mean that is what is really going on.
      non-biased
  • Samsung Galaxy S II

    Last time I checked the Galaxy S II came in several variants. Software alone had Android 2.3 as the original OS, later came Android 2.3.5 and Samsung is supplying an update to 4.0.3 in some countries (Canada was not one of them). Some models had the NFC chip, others didn't, some supported LTE, others either could not support LTE due to chip limitations or the capability was disabled, some have a ARM Cortex-A9 (dual core, 1.2GHz)processor with an Exynos SOC GPU, some have a TI OMAP 4430 (also dual core, 1.2GHz) processor with PowerVR GPU and later versions have a Qualcomm Scorpion (dual core, 1.5GHz) with Qualcomm's Adreno 220 GPU. One report is that T-Mobile in the US of A got yet another variant with a Qualcomm Snapdragon (dual core, 1.5Ghz) processor but no mention of what GPU. Even the quoted screen sizes vary from 4.52 inches to 4.3 inches.

    There just might a good reason as to why it had 3 separate entries in Flurry's list.
    DNSB
    • Re: Samsung Galaxy S II

      So far I haven't seen any bugs caused by differences in SII models. I have seen bugs on the SII that didn't happen on brands, but not between models of the same family. I suppose it's possible but it's not likely.
      Ed Burnette
      • Different but doesn't count?

        So basically you point out in your piece that they are listed separately and that they shouldn't have been yet when pointed out what differences there are in the models your response is I haven't seen bugs between models? So you haven't seen any so there must not be any? Sounds to me at least in this case that Flurry was more accurate than you?
        non-biased
  • Something doesn't add up

    I'm not chiming in to take a shot at Flurry. Instead, I'm chiming in to draw attention to the general iOS favoritism that many in the tech blogosphere exhibit without evidence to back up their bold claims. The truth of the matter, however, is that Android app developers and advertisers understand the media bias and largely ignore it because the FACTS show that Android devs and advertisers are making tremendous strides and boatloads of cash to go along with it. If the numbers you find in "studies" seem dubious, I would advise reading concrete case studies - like how Airpush has grown its mobile ad network so quickly... http://www.airpush.com/blog/redesigning-messaging-relevance-how-we-grew-to-2/
    Messany