Net Market Share vs. StatCounter: Whose online measurements can you trust?

Net Market Share vs. StatCounter: Whose online measurements can you trust?

Summary: If you're curious about which computing platforms are most popular, you can take your pick of two separate independent data sources. What's the difference between the two, and why are some of their numbers so far out of sync?

SHARE:
TOPICS: Browser
29

Measuring the popularity of modern computing platforms is an imperfect science. Nothing makes that point more emphatically than a comparison of the reports from the two top independent data-gathering companies on the web, Net Market Share (also known as Net Applications) and StatCounter.*

On some measures, the two data sources are in near-perfect alignment. On other measures they're wildly far apart. Which one's more accurate? Is it possible they're both right?

I've been looking carefully at these two widely quoted data sources, trying to figure out what makes them tick.

Both companies offer web analytics products that are conceptually similar. By sniffing browser strings and using tracking codes on pages in their respective networks of websites, each firm has assembled a massive store of data about the technology driving those online interactions: country of origin, device type (mobile/desktop/console), operating system, browser, and browser version.

For visits from desktop and portable PCs, the samples are very large for each company. Net Market Share says its reports are based on 40,000 websites worldwide, with 160 million unique visitors per month on desktop and mobile devices. StatCounter’s monthly totals for desktop and notebook PCs consist of 15 billion pageviews recorded at 3 million sites. (That number might sound large, but it’s actually a drop in the bucket, web-wise, given the most recent Netcraft estimate of more than 861 million websites worldwide.)

Net Market Share doesn’t break down its mobile dataset. StatCounter says that in June 2013 its mobile report was based on 2.6 billion pageviews. (Presumably subsequent months have seen similar traffic levels.)

Differences in sample size on a per-country basis have a big influence on usage reports. At StatCounter, for example, more than 26 percent of mobile views are from India, making it the most popular region in its mobile dataset. By contrast, only 7 percent of StatCounter's desktop pageviews come from India. Roughly 21.5 percent of mobile StatCounter's traffic is from the United States; on the desktop, the U.S. share for pageviews is a bit higher, at 24 percent.

Those geographic differences can have a huge influence on more detailed statistical breakdowns. And on mobile devices in particular, different populations have different use cases. In the U.S., a smartphone is likely to be a companion device, with its owner having access to a PC or Mac as well as a tablet. In emerging markets, a mobile device might be an individual's only connection to the online world. So in StatCounter's mobile dataset, with more than a quarter of the traffic drawn from India, it's not surprising that Nokia's Series 40 operating system used on its low-cost Asha line, has a double-digit share according to StatCounter.

Because of these geographic differences and the constantly shifting mix of usage for mobile devices, I'm reluctant to assign too much confidence to either company's mobile reports. The desktop reports, on the other hand, are much more consistent.

Each company has published detailed descriptions of its methodology, which are worth reading:

The crucial difference, as you’ll note if you read the methodologies carefully, is that Net Market Share attempts to measure daily unique users, while StatCounter measures total traffic. If you visit a single page in the Net Market Share network, you’re counted, and then your visits to any other page on any other site in the network are ignored for the rest of the day. Net Market Share weights the data by country, StatCounter doesn’t.

As a result, Net Market Share stats count every user equally, while StatCounter gives extra weight to heavy web users.

Let’s stipulate that the sample sizes for both data sources are sufficiently large, the data collected is accurate, and the populations are reasonably representative of the Internet at large. In that case, any differences you see between the two data sets will represent different behavior (and different choices in hardware and software) on the part of high-volume web users, who are presumably more sophisticated than casual web users. Which measurement is more useful? That depends on why you’re asking the question.

When it comes to the worldwide mix of desktop operating systems, the two companies paint a nearly identical picture of the market:

worldwide-os-share-netmarketshare-statcounter

Both sources calculate the base of Windows PCs worldwide as around 90 percent, give or take a half a percent. Both services peg the worldwide usage of Macs running OS X at a bit over 7 percent. The only noteworthy difference in the other category is that StatCounter detects that 1.3 percent of all desktop/portable PCs are running Android. (Not Chrome OS, which both sources measure at an identical 0.1 percent of usage, but Android. I can’t explain that at all.)

Those numbers line up neatly with years of sales history from IDC and Gartner, which helps their overall air of accuracy.

So with that first group of numbers in total agreement, why do the two sources differ so completely when it comes to desktop browsing? See for yourself—the following two charts summarize the share of usage for top browser versions, as measured by the different companies.

Top-browsers-share-Dec-2013-netmarketshare
Top-browsers-share-Dec-2013-statcounter

If you believe the Net Market Share numbers, Internet Explorer 8 on Windows XP is still the undisputed champ of web browsing. But in the StatCounter world, the latest version of Google Chrome absolutely crushes the competition. What's the difference?

It's possible, even likely, that both of those charts are accurate. As the Net Market Share numbers make clear, most people simply accept the default browser that came with their PC, updating it only if the process is automatic and invisible. Those who use a web browser heavily are more likely to choose a favorite, and that favorite tends to be Chrome.

So, if the question you're asking is "What web browser is the average PC owner most likely to use?" then Net Market Share has the most accurate answer. If, on the other hand, your question is "Which web browsers are you most likely to encounter if you collect a large sample of web traffic?" then StatCounter's numbers are probably going to give you a more meaningful result.

But as I dug into the two datasets, some less obvious conclusions about the state of modern web browsers became apparent. That's a subject for my next post.

* The Wikipedia page that covers Usage share of web browsers includes three other data sources besides Net Market Share and StatCounter:

  • Clicky is a web analytics company that measures pageviews in the style of StatCounter. I don’t have a lot of confidence that their sample is representative of the worldwide market, especially when their “methodology” explanation is a single sentence.
  • W3Counter is a freemium analytics service that claims to track 70,000+ websites worldwide. More than one-third of its traffic comes from the United States, it mixes mobile and desktop traffic with no way to segregate the data, and it counts only the most recent 15,000 visits to each site. Its numbers fluctuate so wildly as to be unbelievable. In July 2013, for example, W3Counter said Internet Explorer’s share of usage was under 22 percent. By October 13 (two weeks before the launch of a new Windows version), IE's share had skyrocketed to more than 29 percent. Four weeks later it was back under 22 percent. Sorry, those numbers don't add up.
  • Wikimedia publishes a consolidated report of traffic to all of the Wikimedia Foundation’s properties, mixing mobile and desktop data. The raw output from its server logs makes for interesting reading but is clearly not representative of the larger world.

Topic: Browser

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

29 comments
Log in or register to join the discussion
  • Dunno about Gartner, though . . .

    "Those numbers line up neatly with years of sales history from IDC and Gartner, which helps their overall air of accuracy."

    LOL.

    As if IDC and Gartner have ever proven themselves accurate.

    Gartner is IMO particularly bad. Frankly, I stop reading most ZDNet articles as soon as it gets mentioned, because I know I'm getting fed bullshit. They're a propaganda house for pushing their pseudo-religious beliefs about where they think the world is headed. They're usually mentioned when somebody wants to proclaim "the cloud" as the ultimate answer for everything.
    CobraA1
    • Gartner

      I've been critical of Gartner analysis and predictions as well, but in this case I am referring to their data-backed quarterly reports of worldwide sales. Those are generally considered quite accurate and reliable.
      Ed Bott
    • Explanation of why numbers differ

      Article not giving the full story.Statcounters given their views on this before.

      Sample size IS important.
      http://gs.statcounter.com/press/open-letter-ms#pageview-sample-size

      Bogus weights make bogus stats.
      http://gs.statcounter.com/press/open-letter-ms#incorrect-weighting

      Pageviews are best metric
      http://gs.statcounter.com/press/open-letter-ms#usage-stats

      Lots of other issues raised too
      http://gs.statcounter.com/press/open-letter-ms#ie-maxthon-bundle
      http://gs.statcounter.com/press/open-letter-ms#differences-netstat
      pauljsmith4
  • Mystified

    I'm not sure how browser market share is an interesting enough metric to get such wide attention. I assume it is just due to its relatively trivial ability to be measured.

    If one is a web developer or markets via the web then you pretty much need to deal with most browsers anyway (yawn). If you are actually a software developer then platform popularity is relevant, but "platform" is not "browser" or even "browser's reported OS." If you relied on that then wide platform differences in the amount of browser vs. smart client usage will mislead you because smart client software use goes entirely uncounted.

    I thought this was already addressed in articles here on inflated iOS numbers due to its users' penchant for spending lots of money on bandwidth-heavy web browsing.
    dilettante
  • No way Chrome accounts for 40% of web traffic...

    I understand the difference in methodology between NetMarketShare & StatCounter... but I simply don't believe that Chrome accounts for 40% of web traffic as the StatCounter numbers suggest.

    I think it either has something to do with the "weighting" that StatCounter admits to doing and/or maybe to some feature in the Chrome software that leads websites to believe it's being visited more often than it is.
    cybersaurusrex
    • Also...

      I've followed the StatCounter website for years now, and they frequently (and drastically) change their methodology. Just take a long-term look at their numbers/charts... and you'll see sudden spikes and valleys several times per year. If their methodology is so good... why such drastic "corrections" so often?
      cybersaurusrex
    • Actually netmarketshare is the outlier regarding browsers

      Statcounter can have higher numbers for chrome, but from Wikimedia to zdnet site own stats, numbers are not so different. Netmarketshare numbers are kinda unique from what I know.
      From people around me, chrome absolute dominate - I would say with a share of 70% or more , even if most people have ie for those "just in case" situations.
      I must say that I'm from Europe and I use opera.
      AleMartin
    • Statcounter seems to be vulnerable to bot manipulation

      Or maybe they count Google web crawling as Chrome. Either way I scoff at their browser numbers.
      greywolf7
    • Chrome preloads

      Chrome preload pages, which you most likely never view, which is why statcounter has it crushing every other browser.
      You open one page and chrome can go and fetch 10 others before you even click on any links.
      It does this by default and preloads much more than other browsers and most people never even know its chewing thru extra bandwidth.
      warboat
      • StatCounter made changes to remove the impact of pre-loading

        I don't remember the details, and am happy to wait for Ed to discuss it (likely to come up in his next article), but StatCounter followed Net Market Share's lead, a year or so after them, in attempting to remove such pre-loads from consideration. No idea of the efficacy of the approach that either took.
        daboochmeister
        • pre-fetch headers

          Browsers like Firefox and Safari and IE sends a special header for pre-fetched content.
          Chrome doesn't send any pre-fetch indicator so the web server cannot distinguish a Chrome pre-fetch from an actual view.
          This is the reason for Chrome crushing statcounter.
          warboat
          • Again, wait for Ed, he will probably cover ...

            ... but my recollection is that both Net Market Share and StatCounter detailed exactly how they distinguish Chrome fetches, and that they DID in fact have a method.
            daboochmeister
    • Doesn't

      Chrome also cache links on the visited page, so that they load quicker? If you visit a Stat page, and it has 20 links, they'll get 20 more hits from you, if you are using Chrome, if you are using another browser without forward link caching, then they'll get 1 hit.
      wright_is
    • Chrome users more savvy?

      The argument for the unusually high Chrome browser numbers with StatCounter seems to be that Chrome users are more savvy (since they chose Chrome over the default IE) and, therefore, use the internet way more than everyone else.

      Here's why that argument falls flat. If savvy internet users choose non-IE browsers (which has not been proven), then why don't savvy Firefox users also own 40% of the market like Chrome users? Just as many people use Firefox as Chrome... and yet Chrome is steamrolling them as well. Sorry, no way.

      The reality is that at least half of the desktops out there are in the enterprise, and I'd estimate that close to 80% of those desktops are running Internet Explorer... for 8 hours a day. So for StatCounter's browser numbers to be correct, Chrome users (about 20% of desktop users) would have to rush home and use Chrome all night long (and even while sleeping) just to equal all of those PCs running IE during the workday... and that doesn't even include the 50% of home PCs also running IE.

      In short, StatCounter's browser numbers are a joke, and likely due to something in the Chrome software that makes it counted 10x more than it actually should.
      cybersaurusrex
  • how does use of extensions change the results?

    for instance , extensions that change the browser id's
    or safety extensions like NoScript ?
    chips@...
  • Net Market Share vs. StatCounter: Whose online measurements can you trust ?

    Strange you should be asking this question Ed as many of the articles you write are based around such statistics.

    Now it would seem you are questioning the validity of them.
    5735guy
  • Uh, no

    I am explaining how they work. Did you not read?
    Ed Bott
    • A leading question Ed....

      Do you trust them?

      YES OR NO
      5735guy
  • I find....

    StatCounter's browser results laughable. Yes. I know people who use it, but not at nearly 40%. W3Counter's information has always been full of dung [aside from the example given].
    [BTW, IE8 was available also on Vista and Windows 7 - not just Windows XP.]
    Gisabun
  • How are automated processes (e.g. WinUpdate) handled?

    Most automated processes that access the web on behalf of a user on Windows boxes end up using an IE control under the covers as their HTTP library. Given Net Market Share's approach, if such an automated process was counted as a "visit" against one of the sites they track, and the process ran at e.g. 12:01am every night, then even if the user on that PC used Chrome for ALL of his/her interactive browsing, they would never be counted as a Chrome user.

    Obviously, they know this, and I assume they've made effort to remove such a bias (e.g., I doubt they include WinUpdate calls as "visits", that's too obvious) - but even if just some of the automated processes are missed, it could easily skew the statistics significantly.

    The same set of processes need to be understood in the context of the StatCounter methodology, of course ... but since the "first visit is the only one that counts" approach isn't used, it would seem less prone to skew.

    Thoughts?
    daboochmeister