Weird science: How shaky web usage reports distort our view of tech markets
The monthly reports of browser and OS usage from two web analytics firms have taken on a life of their own.
It’s easy to create horse-race narratives from these first-of-the-month data drops. But in recent news cycles these numbers have been treated as gospel rather than highly variable and occasionally inconsistent data points.
This month’s reports from Net Applications (NetMarketShare) and StatCounter illustrate just how strange the picture can get if you take the data at face value. (If you’re unfamiliar with how these two products work, here’s an explainer I wrote earlier this year: “Net Market Share vs. StatCounter: Whose online measurements can you trust?”)
My colleague Jack Schofield read the latest NetMarketShare numbers from Net Applications (.com) and noted these odd spikes in the latest data:
He posted this skeptical report over the weekend:
Windows XP's market share has tumbled to just 17.2 percent — a fall of 6.7 percentage points — according to Netmarketshare's latest figures, published on Saturday by Net Applications.
At the same time, the combined market share of Windows 8 and Windows 8.1 has climbed to 16.8 percent, which is only 0.4 percentage points behind the venerable XP.
Perhaps next month's results will show that XP hasn't really fallen so far so fast, and the October numbers are a statistical blip.
As it turns out, this was something more than a “statistical blip.”
After those initial numbers appeared, Net Applications quickly published a notice on its site that it was reviewing the data. An update explained the sudden and dramatic swing:
NetMarketShare has recently shown a large drop in Windows XP share from September to October. This drop was primarily caused by a major change in the network of sites we have in China. A group of large Chinese publishers with a very large number of visitors per day had audiences heavily skewed towards Windows XP (nearly 100% XP). In researching the nature of the sites, we determined they were not appropriate for our network. We removed those publishers ourselves, which caused the shift since Chinese traffic is weighted higher due to lower coverage. The current data set is more accurate than in the past due to this.
Featured
That explanation, while perfectly logical, also illustrates why these statistics need to come with boldface, banner-size disclaimers. From month to month, there’s no guarantee that the sample data is reasonably representative of the Internet at large. In fact, a large enough group of unrepresentative sites that are part of the analytics network can skew the results.
That’s exactly what’s been happening at NetMarketShare, which also cryptically noted that it plans to begin publishing data that will show what it calls “fraudulent traffic” from bots, toolbars, and other automated methods.
Apparently XP wasn't that popular after all. We'll never know, because Net Applications says it can't retroactively adjust previous reports.
In a separate incident, I noted an odd finding in this month’s data from StatCounter. Here are the results for desktop browser usage from the United States, covering the 13 months ending on October 31, 2014:
I’ve been skeptical of drawing conclusions from these reports previously, given their wide swings that are unrelated to external events. This one is really a head-scratcher.
We can all construct our own narratives for the battle between Internet Explorer and Chrome, and we can explain the relative positions of Safari and Firefox easily enough.
But what possible explanation is there for the 360 Safe browser, developed by the Chinese Web giant Qihoo, to be consistently in 5th place on that list? According to StatCounter, this browser has been responsible for an average of 2.68 percent of all web traffic from the United States for the eight months starting in March 2014.
That didn’t sound right to me, so I asked StatCounter for an explanation. Here’s what they replied:
We completely understand Ed's gut reaction - the presence of 360 Safe browser in the US does look unusual. We were similarly skeptical back in 2012 when we first noticed the browser appear in the US stats. Despite our investigations back then (and again now in response to Ed's email), we have not identified anything untoward in relation to the presence of this browser in the US - US-based IP addresses are using this browser hence its presence in the stats.
According to StatCounter, this growth has been “very gradual and appears ‘normal.’” They first noted the appearance of the 360 Safe browser in 2013, coincident with the launch of an international version of the browser as well as an English-language homepage.
Qihoo 360 Technology, which is a public company traded on the New York Stock Exchange (QIHU), claims to have hundreds of millions of users globally. A StatCounter spokesperson notes that the company set up a US office in 2013, added a venture funding arm in Silicon Valley this year, and have a business development team for North America.
And maybe their success has been because their customers are unlikely to identify themselves publicly:
A large portion of the US 360 traffic is visiting "adult" sites. This cohort of web users may have significant privacy concerns. We feel that, perhaps, 360 is becoming a popular choice for these users because it claims to offer very secure browsing which filters phishing attempts and malicious URLs and for which additional free security services are available.
U.S.-centric tech journalists love to write about the fortunes of American companies trying to enter the Chinese market. If StatCounter’s data is accurate, then maybe Lenovo isn’t the only Chinese company making inroads in North America.