Why you should be skeptical of Chitika's market-share reports
Chitika, an online advertising company, has been remarkably successful lately at getting top-tier tech sites to quote its "research reports." But a closer look at this company suggests that maybe journalists should be a lot less trusting of its data.
I have no idea how effective Chitika is in their role as an online ad network. But they've been a smashing success at getting the media to uncritically report their increasingly frequent "research reports."
For the past several years, Chitika has been churning out confident reports chock full of data and feeding them to a hungry tech press. The strategy has paid off for this Massachusetts-based online ad network, which has been in business for nearly a decade. Their reports have been approvingly quoted in a Who's Who of tech blogs, including TechCrunch, AllThingsD, the Los Angeles Times, and even ZDNet. Those publications have reprinted Chitika's sweeping conclusions about the adoption rate of iOS versions, relative market shares of Android and iOS devices, and adoption rates for tablets, to name just a few.
Earlier today, our own Adrian Kingsley-Hughes passed along a Chitika report that claimed "the iPad's Web usage share fell by a dramatic 7.1 percentage points after December 25." And according to Techmeme eight other tech sites reported the same story, citing that number as an unquestioned fact.
Really? In what universe does that kind of shift in market share happen literally overnight? Hint: not this one.
I love data-driven journalism more than just about anyone, but Chitika's numbers don't pass the sniff test. I've avoided writing stories based on Chitika's many press releases precisely because I couldn't find any evidence that the numbers were reliable, trustworthy, consistent, or meaningful. On the contrary, the numbers have frequently been just laughably inconsistent.
For example, last May Chitika breathlessly reported that OS X usage was up 2.58% month over month (from 11.44% to 14.02%), while Windows usage was down 2.99%. There was no reason for the dramatic shift; it just happened. But one month later Windows usage increased dramatically while Mac usage was down significantly. Meanwhile, in the month of July, Chitika reported that half of all Linux users abruptly stopped using their preferred operating system, with Linux share dropping from 2.05% to 1.12%.
Really? Really? Those sorts of extreme variations suggest that the underlying data is about as accurate as a daily horoscope.
This morning I fired off a note to Chitika asking if they could provide details about their methodology. I got an immediate response from a press contact who pointed me to a page containing this terse description:
Chitika Insights is the independent research arm of online ad network Chitika. Our team of passionate and innovative Data Solutions Engineers produce a series of reports each week uncovering unique trends in Internet usage. Research studies are conducted by tapping into Chitika’s extensive network comprised of over 250,000 sites, totaling hundreds of millions of ad impressions each week. The sample size used for all Chitika Insights studies is large and diversified, ensuring an unbiased sample and allowing us to draw greater conclusions as to what is happening on the Internet across North America.
That's not exactly what I was looking for. As any statistician could tell you, simply having a large sample size doesn't mean you get valid conclusions. Garbage in, garbage out. Doubling or tripling the amount of data just makes for a bigger pile of garbage.
I replied with a request for more details about the data set and about the qualifications of the people assembling these reports. (Is there a trained statistician in the house?) Chitika has not yet replied.
Meanwhile, a little poking around at Chitika's website makes me even more skeptical of the quality of the company's research. The Publishers page, for example, where a website owner can sign up to run Chitika ads, contains testimonials from supposedly satisfied customers. I refreshed the page repeatedly and built up a list of 30 domains offering glowing testimonials about their success with Chitika's ad network.
Of those 30 sites, 10% resulted in 404 or 401 errors. Another 20% led to domains that were parked with a registrar and serving up nothing but spammy links. Yet another 20% were redirected to domains other than the original, or went to crappy content farms or make-money-fast sites.
Here, for example, is a Chitika testimonial from one supposedly satisfied customer:
But this is what you see if you visit that website:
Yes, that site is parked at a domain host, and there's not a Chitika ad in sight. It's a pattern I saw over and over again when I checked out those testimonials.
In my investigation, half of the web sites that Chitika promotes as success stories either don't exist or exist exclusively to serve ads. The remaining 50% appeared generally weak and sad. Several sites hadn't been updated in months or years, and only a handful looked like they represented serious ongoing businesses.
As a potential advertiser, I would not be impressed. As a journalist, I wonder whether the same sloppiness exhibited on this promotional page extends to the company's research.
The FTC reached a settlement with online advertising company Chitika, Inc. that ends the company’s allegedly deceptive practice of tracking consumers’ online activities even after they have chosen to opt out of online tracking on Chitika’s website.
According to the FTC complaint, from at least May 2008 through February 2010, Chitika’s opt-out lasted only 10 days. After that time, Chitika placed tracking cookies on browsers of consumers who had opted out and targeted ads to them again. The FTC charged Chitika’s claims about its opt-out mechanism were deceptive and violated federal law.
Put all those pieces together and you don't get a picture of a company whose data should be trusted on its face. Yet big tech news sites (including ZDNet) routinely do exactly that, and in the process they give favorable publicity to a company that does not deserve it.
Maybe the next time you read a report based on data from Chitika, you could send the reporter a link to this post and ask if they've got a second source.
Update, January 4, 8AM PST. Chitika responds in the Talkback section:
A response from Chitika Insights
Chitika Insights, the independent research arm of online at network Chitika, has several quality checks in place that ensure accuracy and reliability of our data. This includes comparing a sample of our network with overall search traffic for each study released. Second, we collaborate with manufactures [sic] on a regular basis to review market share data seen on our network and ensure accuracy. Third, we touch base directly and indirectly with other third party market analysts to confirm our numbers are in fact in sync with the industry, validating our data. The research team at Chitika Insights includes trained statisticians and data scientists who follow a rigorous process of analyzing and validating the accuracy of each study before release. We plan to address any further questions from Mr. Bott when we speak with him later this morning.
Update: Following the publication of this postI had a (mostly) pleasant chat with executives from Chitika as well as the person in charge of their data. There is no question in my mind the company is sincere, and they acknowledged mistkaes in their marketing. I heard nothing that convinced me their data is reliable nor thatdo I believe it can be generalized to the Web at large. - EB