Why search engines are about more than just results
Back when Windows 7 was in beta, I was in such a hurry to start using a new PC that I forgot to switch the search engine to Google; and it was three days before I actually noticed that the logo on top of the page was Bing.
That was three days of getting the results I needed, that I thought only Google would produce; but I was searching for technical details about Windows XP, information about crafts and recipes, and other things that might not match what you need.
And as with Google, the longer you use Bing, the more relevant the results you get because Bing learns from the links you click; I see slightly better results from Bing when I'm logged in.
That's why the row around Microsoft's 'Bing it on' campaign has intrigued me. Microsoft has claimed that its research shows that people chose Bing web search results over Google by a factor of nearly two to one in blind comparison tests. But Ian Ayres, a Yale law professor who blogs at Freakonomics did his own research and came up with a different result — according to his research, most participants preferred Google search results to Bing search results (53 percent to 41 percent).
New from Google
It's worth pointing out that the two surveys operated differently: Ayres team paid Mechanical Turk users 40 cents to take the Bing It On challenge (the team didn't get enough users by offering 40 cents so they had to increase the payment to $1 to get 985 users).
The Bing study (done by a research company) also had close to 1,000 people but they didn’t know Microsoft was involved in the study or what search engines they were comparing. And the Mechanical Turk users were much more likely to be white, male, liberal, non-religious and to have a degree than the mainstream US population. Ayres' own analysis shows that the types of people who are under-represented in his sample also tend to be the type of people who "were statistically more likely to prefer Bing".
The Freakonomics survey didn't use the same searches as the Bing study either. The Bing It On site lets you type your own search or suggests recent popular searches from Bing in case you can't think of something to search for.
But while one Bing study of 1,000 people let users type in their own search (in that study 57 percent of users prefered Bing, 30 percent preferred Google), another Microsoft study that Ayres criticises gave the 1,000 participants their choice of several random searches from Google's Zeitgeist survey. Both did ten searches in total; Bing It On lets you do five, so that's what Ayres' Turkers did too. Ayres added another option; for a third of the Turkers, he pre-selected what the search should be.
All of this complicated analysing the results.
But then again, maybe Bing has got worse — or Google got better — since the original Bing study, which was back in 2012?
Certainly Google has started using a lot of Bing's layout tricks like putting images in the main search results in case that's a better way of getting the information you need.
But since Ayres did his study, Microsoft has released the results of another 1,000-user blind study it did in the UK; this time the ten searches each came from the Google 2012 Zeitgeist list and from Google's June 2013 trends list. This time, 53 percent picked Bing as having the best results, 34 percent preferred Google and 13 percent said they were as good as each other. That's remarkably close to the 52 percent Bing, 36 percent Google and 12 percent 'both are good' results from the second US study.
In the end, the best search engine for you is the one that gives you useful results, but the way you feel about a search engine isn't always just about the results. It might be habit or brand loyalty or liking the page layout more or just the expectation that one is going to be better.
In the example above, given the level of animus against Microsoft felt by some, simply telling people they're testing Bing could be enough to skew the results, especially when it's trivially easy to tell which is which. In fact, Survey Monkey found exactly that in a recent study; Bing's results scored better than Google's — as long as they labelled them as coming from Google. When they marked Google results as coming from Bing, they were less popular.
For me, a blind test like the original Bing trial is the best way to find out but even the take-it-yourself Bing It On challenge is a good way to find out if it's Google the brand or Google's results you actually like.
Just remember that your test and your opinion isn't a statistic; it's just an anecdote and 299 more of those only give you anecdata.