Why search engines are about more than just results

Why search engines are about more than just results

Summary: It's not just what you find, but how you feel about the brand that matters when it comes to web search - which is why comparing different search engines is more complicated than who clicks on what.

SHARE:

Back when Windows 7 was in beta, I was in such a hurry to start using a new PC that I forgot to switch the search engine to Google; and it was three days before I actually noticed that the logo on top of the page was Bing.

That was three days of getting the results I needed, that I thought only Google would produce; but I was searching for technical details about Windows XP, information about crafts and recipes, and other things that might not match what you need.

And as with Google, the longer you use Bing, the more relevant the results you get because Bing learns from the links you click; I see slightly better results from Bing when I'm logged in.

That's why the row around Microsoft's 'Bing it on' campaign has intrigued me. Microsoft has claimed that its research shows that people chose Bing web search results over Google by a factor of nearly two to one in blind comparison tests. But Ian Ayres, a Yale law professor who blogs at Freakonomics did his own research and came up with a different result — according to his research, most participants preferred Google search results to Bing search results (53 percent to 41 percent).

It's worth pointing out that the two surveys operated differently: Ayres team paid Mechanical Turk users 40 cents to take the Bing It On challenge (the team didn't get enough users by offering 40 cents so they had to increase the payment to $1 to get 985 users).

The Bing study (done by a research company) also had close to 1,000 people but they didn’t know Microsoft was involved in the study or what search engines they were comparing. And the Mechanical Turk users were much more likely to be white, male, liberal, non-religious and to have a degree than the mainstream US population. Ayres' own analysis shows that the types of people who are under-represented in his sample also tend to be the type of people who "were statistically more likely to prefer Bing".

The Freakonomics survey didn't use the same searches as the Bing study either. The Bing It On site lets you type your own search or suggests recent popular searches from Bing in case you can't think of something to search for.

But while one Bing study of 1,000 people let users type in their own search (in that study 57 percent of users prefered Bing, 30 percent preferred Google), another Microsoft study that Ayres criticises gave the 1,000 participants their choice of several random searches from Google's Zeitgeist survey. Both did ten searches in total; Bing It On lets you do five, so that's what Ayres' Turkers did too. Ayres added another option; for a third of the Turkers, he pre-selected what the search should be.

All of this complicated analysing the results.

But then again, maybe Bing has got worse — or Google got better — since the original Bing study, which was back in 2012?

Certainly Google has started using a lot of Bing's layout tricks like putting images in the main search results in case that's a better way of getting the information you need.

But since Ayres did his study, Microsoft has released the results of another 1,000-user blind study it did in the UK; this time the ten searches each came from the Google 2012 Zeitgeist list and from Google's June 2013 trends list. This time, 53 percent picked Bing as having the best results, 34 percent preferred Google and 13 percent said they were as good as each other. That's remarkably close to the 52 percent Bing, 36 percent Google and 12 percent 'both are good' results from the second US study.

In the end, the best search engine for you is the one that gives you useful results, but the way you feel about a search engine isn't always just about the results. It might be habit or brand loyalty or liking the page layout more or just the expectation that one is going to be better.

In the example above, given the level of animus against Microsoft felt by some, simply telling people they're testing Bing could be enough to skew the results, especially when it's trivially easy to tell which is which. In fact, Survey Monkey found exactly that in a recent study; Bing's results scored better than Google's — as long as they labelled them as coming from Google. When they marked Google results as coming from Bing, they were less popular.

For me, a blind test like the original Bing trial is the best way to find out but even the take-it-yourself Bing It On challenge is a good way to find out if it's Google the brand or Google's results you actually like.

Just remember that your test and your opinion isn't a statistic; it's just an anecdote and 299 more of those only give you anecdata.

Related stories

Topics: Microsoft, Emerging Tech, Tech Industry

Mary Branscombe

About Mary Branscombe

Mary Branscombe is a freelance tech journalist. Mary has been a technology writer for nearly two decades, covering everything from early versions of Windows and Office to the first smartphones, the arrival of the web and most things inbetween.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

3 comments
Log in or register to join the discussion
  • Search Engines and money

    Way back when, when the public internet first got going with everyone creating "web sites", Microsoft, Netscape, Yahoo, AOL and even Google all wondered how they were going to make money off of the potentially billions of people viewing the World Wide Web. Microsoft, Netscape, Yahoo, AOL all were convinced that businesses would pay for placement on search engines. The result was whomever paid the most ended up on the first page of the search results. Most often, it was a business selling a product, not necessarily information on the search keywords. One often had to scroll through pages and pages of search results to find information, rather than someone selling a product. Then there was the pay per click businesses had to pay each time someone clicked the search engine link.

    Google on the other hand sold ad space. One could search and get results that were not pages and pages of businesses selling products.

    So what are the potentially billions of people going to do? Are they going to stay with a search engine that only gives them thousands of businesses selling products and not the information they were looking for? or go to another search engine that gives them the information they search for, and off on the margins, see ads spots? Google is so successful financially because they sold ad space, not search placement and everyone uses Google to search because they can find the information, just pages of businesses selling a product.

    Side Note: video on the web. At one time I felt video on the web was the future. But as time went on, I found I would rather read, that watch. Click a link for information, and up pops a video, I find I close the page because I don't want to watch a news video on the web, I want to read to get the information. That's one issue with Yahoo that makes me not use it's site. An interesting headline is there, click the link and there's a video with an ad, before I even get to the information I was looking for.
    CG IT
  • Search Engines and money

    Way back when, when the public internet first got going with everyone creating "web sites", Microsoft, Netscape, Yahoo, AOL and even Google all wondered how they were going to make money off of the potentially billions of people viewing the World Wide Web. Microsoft, Netscape, Yahoo, AOL all were convinced that businesses would pay for placement on search engines. The result was whomever paid the most ended up on the first page of the search results. Most often, it was a business selling a product, not necessarily information on the search keywords. One often had to scroll through pages and pages of search results to find information, rather than someone selling a product. Then there was the pay per click businesses had to pay each time someone clicked the search engine link.

    Google on the other hand sold ad space. One could search and get results that were not pages and pages of businesses selling products.

    So what are the potentially billions of people going to do? Are they going to stay with a search engine that only gives them thousands of businesses selling products and not the information they were looking for? or go to another search engine that gives them the information they search for, and off on the margins, see ads spots? Google is so successful financially because they sold ad space, not search placement and everyone uses Google to search because they can find the information, just pages of businesses selling a product.

    Side Note: video on the web. At one time I felt video on the web was the future. But as time went on, I found I would rather read, that watch. Click a link for information, and up pops a video, I find I close the page because I don't want to watch a news video on the web, I want to read to get the information. That's one issue with Yahoo that makes me not use it's site. An interesting headline is there, click the link and there's a video with an ad, before I even get to the information I was looking for.
    CG IT
  • Agreed

    Never use Yahoo for that very reason. Never use Bing or Google either. I liked Ask from day 1 and will not change whilst it consistently gives me everything I need.
    Snak