Why you should be skeptical of Chitika's market-share reports

Why you should be skeptical of Chitika's market-share reports

Summary: Chitika, an online advertising company, has been remarkably successful lately at getting top-tier tech sites to quote its "research reports." But a closer look at this company suggests that maybe journalists should be a lot less trusting of its data.

SHARE:

I have no idea how effective Chitika is in their role as an online ad network. But they've been a smashing success at getting the media to uncritically report their increasingly frequent "research reports."

For the past several years, Chitika has been churning out confident reports chock full of data and feeding them to a hungry tech press. The strategy has paid off for this Massachusetts-based online ad network, which has been in business for nearly a decade. Their reports have been approvingly quoted in a Who's Who of tech blogs, including TechCrunchAllThingsD, the Los Angeles Times, and even ZDNet. Those publications have reprinted Chitika's sweeping conclusions about the adoption rate of iOS versions, relative market shares of Android and iOS devices, and adoption rates for tablets, to name just a few.

Earlier today, our own Adrian Kingsley-Hughes passed along a Chitika report that claimed "the iPad's Web usage share fell by a dramatic 7.1 percentage points after December 25." And according to Techmeme eight other tech sites reported the same story, citing that number as an unquestioned fact.

Really? In what universe does that kind of shift in market share happen literally overnight? Hint: not this one.

I love data-driven journalism more than just about anyone, but Chitika's numbers don't pass the sniff test. I've avoided writing stories based on Chitika's many press releases precisely because I couldn't find any evidence that the numbers were reliable, trustworthy, consistent, or meaningful. On the contrary, the numbers have frequently been just laughably inconsistent.

For example, last May Chitika breathlessly reported that OS X usage was up 2.58% month over month (from 11.44% to 14.02%), while Windows usage was down 2.99%. There was no reason for the dramatic shift; it just happened. But one month later Windows usage increased dramatically while Mac usage was down significantly. Meanwhile, in the month of July, Chitika reported that half of all Linux users abruptly stopped using their preferred operating system, with Linux share dropping from 2.05% to 1.12%.

Really? Really? Those sorts of extreme variations suggest that the underlying data is about as accurate as a daily horoscope.

This morning I fired off a note to Chitika asking if they could provide details about their methodology. I got an immediate response from a press contact who pointed me to a page containing this terse description:

Chitika Insights is the independent research arm of online ad network Chitika. Our team of passionate and innovative Data Solutions Engineers produce a series of reports each week uncovering unique trends in Internet usage. Research studies are conducted by tapping into Chitika’s extensive network comprised of over 250,000 sites, totaling hundreds of millions of ad impressions each week. The sample size used for all Chitika Insights studies is large and diversified, ensuring an unbiased sample and allowing us to draw greater conclusions as to what is happening on the Internet across North America.

That's not exactly what I was looking for. As any statistician could tell you, simply having a large sample size doesn't mean you get valid conclusions. Garbage in, garbage out. Doubling or tripling the amount of data just makes for a bigger pile of garbage.

I replied with a request for more details about the data set and about the qualifications of the people assembling these reports. (Is there a trained statistician in the house?) Chitika has not yet replied.

Meanwhile, a little poking around at Chitika's website makes me even more skeptical of the quality of the company's research. The Publishers page, for example, where a website owner can sign up to run Chitika ads, contains testimonials from supposedly satisfied customers. I refreshed the page repeatedly and built up a list of 30 domains offering glowing testimonials about their success with Chitika's ad network.

Of those 30 sites, 10% resulted in 404 or 401 errors. Another 20% led to domains that were parked with a registrar and serving up nothing but spammy links. Yet another 20% were redirected to domains other than the original, or went to crappy content farms or make-money-fast sites.

Here, for example, is a Chitika testimonial from one supposedly satisfied customer:

chitika-oneinchgroup-testimonial

But this is what you see if you visit that website:

chitika-oneinchgroup-website

Yes, that site is parked at a domain host, and there's not a Chitika ad in sight. It's a pattern I saw over and over again when I checked out those testimonials.

In my investigation, half of the web sites that Chitika promotes as success stories either don't exist or exist exclusively to serve ads. The remaining 50% appeared generally weak and sad. Several sites hadn't been updated in months or years, and only a handful looked like they represented serious ongoing businesses.

As a potential advertiser, I would not be impressed. As a journalist, I wonder whether the same sloppiness exhibited on this promotional page extends to the company's research.

And then there's this FTC news release from March 14, 2011:

The FTC reached a settlement with online advertising company Chitika, Inc. that ends the company’s allegedly deceptive practice of tracking consumers’ online activities even after they have chosen to opt out of online tracking on Chitika’s website.

[...]

The FTC alleged that in its privacy policy the company says that it collects data about consumers’ preferences, but allows consumers to opt out of having cookies placed on their browsers and receiving targeted ads. The privacy policy includes an “Opt-Out” button. Consumers who click on it activate a message that states, “You are currently opted out.”

According to the FTC complaint, from at least May 2008 through February 2010, Chitika’s opt-out lasted only 10 days. After that time, Chitika placed tracking cookies on browsers of consumers who had opted out and targeted ads to them again. The FTC charged Chitika’s claims about its opt-out mechanism were deceptive and violated federal law.

Put all those pieces together and you don't get a picture of a company whose data should be trusted on its face. Yet big tech news sites (including ZDNet) routinely do exactly that, and in the process they give favorable publicity to a company that does not deserve it.

Maybe the next time you read a report based on data from Chitika, you could send the reporter a link to this post and ask if they've got a second source.

Update, January 4, 8AM PST. Chitika responds in the Talkback section:

A response from Chitika Insights

Chitika Insights, the independent research arm of online at network Chitika, has several quality checks in place that ensure accuracy and reliability of our data. This includes comparing a sample of our network with overall search traffic for each study released. Second, we collaborate with manufactures [sic] on a regular basis to review market share data seen on our network and ensure accuracy. Third, we touch base directly and indirectly with other third party market analysts to confirm our numbers are in fact in sync with the industry, validating our data. The research team at Chitika Insights includes trained statisticians and data scientists who follow a rigorous process of analyzing and validating the accuracy of each study before release.  We plan to address any further questions from Mr. Bott when we speak with him later this morning.

Update: Following the publication of this postI had a (mostly) pleasant chat with executives from Chitika as well as the person in charge of their data. There is no question in my mind the company is sincere, and they acknowledged mistkaes in their marketing. I heard nothing that convinced me their data is reliable nor thatdo I believe it can be generalized to the Web at large. - EB

Topics: iPad, Android, Apple, Browser, Linux, Windows

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

31 comments
Log in or register to join the discussion
  • Press release journalism

    Reporters should check out alleged facts supplied by self-interested corporations and pressure groups (at least get second opinions from sources they trust), but both of us know they often don't. That's why companies like Chikita are able to plant propaganda with relative ease. Fortunately, reprinted press releases are relatively easy to spot if you've read enough of the raw material. Other sorts of propaganda are more insidious because they're harder to detect (making it important to get one's news from multiple sources).
    John L. Ries
    • not propaganda

      I don't believe this is propaganda. It's simply inconsistent to be.

      But it is an good example of cheap attempts to get into the limelight.
      danbi
    • All of these market share numbers are bogus...

      I don't trust any of these market share metrics because they all have wildly different numbers. StatCounter still has IE at about 50% of the browser market, but NetApplications has IE at about 25%. Now... how can they both be accurate?

      Furthermore, even though we know that Android phones are outselling iPhones by about 3-to-1, these market share companies show that the iPhone has double the market share. Why? Because they measure hits on websites. But that doesn't change the reality.

      The interesting thing is that, no matter how these market share companies decide to measure things, it always seems to favor Apple somehow... which is totally suspicious.

      In short, all of these numbers are bogus.
      newyorkcitymale
  • Honestly

    What is harder to believe, that the other platforms lead in market share but failed to keep up on internet usage or that the numbers now have a 7+ point swing in the other direction?

    Reality is, I don't believe the iOS web usage numbers were ever that high and somebody else missed the mark. If anything I would say the numbers are still too high in favor of iOS or, they are measuring websites like Apple.com!
    slickjim
  • I've been skeptical of this source as well

    I've never heard of Citika before blogs started picking up their report in surface figures. I found it interesting that as opposed to most other sites which just provide data, Chitika spends a considerable effort in spinning the data and packaging it into attention grabbing headlines. They've hit Windows 8, Surface, iPad, and Windows Phone... probably others... with some outrageous headlines.

    What's worse is I don't even see where they make the raw data available, like Hitslink or Statcounter.
    ModernMech
    • Indeed

      I had never heard of them as well.

      BTW, what are your thoughts on MS retiring the Mesh, ED? It was one of the best MS software and it seems a big mistake on their part. Am I missing something?
      Undisc
      • Well if you are not looking for direct PC to PC synchronization

        you could use via SkyDrive integration. That's what Microsoft is also saying about Windows LiveMesh to SkyDrive transition. I am using it and I miss the facility of bypassing cloud, but this gives me more flexibility. And there is SugarSync, which also provide the same facilities of SkyDrive.
        Ram U
        • Peer-to-peer sync

          Peer-to-peer sync is the feature I'll miss the most.
          Undisc
  • Another excellent analysis by Ed, thank you!

    Zdnet should appoint you to train some of the "other" Zdnet writers on how to be professional and objective.
    ninjacut
    • Thanks for the comment

      I need a laugh.
      RickLively
  • Why you should be skeptical of Chitika's market-share reports

    Because Loverock Davidson had a hand in it.......
    Over and Out
  • Chitika ...

    OK, Chitika goes into the same barrel down by the river that contains DigiTimes.
    John Baxter
  • Good catch Ed

    It seems those people just seek cheap popularity.

    I am recently curious about this "testmonials" phenomena... does anyone take these seriously? It seems, that at least in this case the testimonials are either not verified, or just generated from a list of names. Doesn't speak much of the quality of their data, if they don't even verify who is "endorsing" them.

    Like I mentioned in the AKH's blong post - data interpretation is not for everyone.
    danbi
  • A response from Chitika Insights

    Chitika Insights, the independent research arm of online at network Chitika, has several quality checks in place that ensure accuracy and reliability of our data. This includes comparing a sample of our network with overall search traffic for each study released. Second, we collaborate with manufactures on a regular basis to review market share data seen on our network and ensure accuracy. Third, we touch base directly and indirectly with other third party market analysts to confirm our numbers are in fact in sync with the industry, validating our data. The research team at Chitika Insights includes trained statisticians and data scientists who follow a rigorous process of analyzing and validating the accuracy of each study before release. We plan to address any further questions from Mr. Bott when we speak with him later this morning.
    Chitika Insights
    • What a hopeless rebuttal

      No numbers whatsoever, just qualitative and vague language. Total spin.
      Rurik Bradbury
    • Nothing you just said...

      Refutes the contention that your analytic conclusions are derived from garbage data culled from dubious sources. The accuracy of you conclusions might be sound in terms of a classroom exercise on analyzing a particular set of data, but that set of data is just worthless, ergo so are meaningful conclusions.
      jvitous
      • Its the old story with IT of any kind...

        ...garbage in ...garbage out.
        Cayble
  • Article points to TABLOIDISM

    ... Ed, well written and informative article - thank you. Sounds like this vendor is a little bit into the tabloid side of things, kind of like when one week one celebrity is overweight, the next week they're skin and bones, the week after that they were visited by bat creatures from Mars, etc.

    Again, well researched and written.
    rock06r
  • Journalists, bloggers and troll bloggers

    Reports like this are precious to troll bloggers because the content has the potential to promote religious wars amongst the most rabid and nutcase techies:

    o Apple fans are dismayed that their religion is losing control
    o Android (and Linux) fans are energized as the future clearly belongs to their religion
    o Microsoft fans are outraged because their religion is but a footnote in the report
    o Or the smarter ones among them argue that the underlying data and associated information are not available and , therefore, the report should be considered suspect (besides, their religion is winning anyway)

    With this particular Chitika report, there's something for everyone to fight over. The almighty 'clicks' from these religious wars translate to advertising $.
    Rabid Howler Monkey
    • Not sure of your conclusion...

      ..all the article shows is that the statistics Chitika works with appear to come from suspect sources that provide for questionably odd results.

      That can reasonably lead one to conclude if this is indeed the case the results from their compilation and analysis of those statistics may be leading to inacurate results.

      While the results themselves may lead to a flame war amongst the stupid its not Ed who compiled the results of Chitikas reports. But there is no question its important for people to know what they are reading if they find themselves reading about some report from Chitika. Positive or negative, I wouldnt be putting much faith in any of their results unless and until I hear of some better explaination for the radical upswings and downswings in market share they are claiming. It does sound bizarre.

      I dont think such an article itself should ever reasonably produce a flame war because the article basically says "none of these Chitika results may have any substantive reliability given their origin and wild swings". That shouldnt produce any flame war among the rational. Just a war for the stupid.
      Cayble