Will Google's blind faith in the algorithm doom its future?

Will Google's blind faith in the algorithm doom its future?

Summary: Google search has been having a tough year. It's Panda update has had a difficult time targeting content farms and has accidentally affected a lot of good stuff.


Google's search engine is a triumph of technology. There's no denying that.

It was the capstone that completed the initial structure of the Internet. But, the Internet is now in the midst of a dramatic remodel and it's unclear whether Google search will get the refresh it needs to make it more appealing than ever or if it will be one of the things that gets painted over.

Photo credit: iStockPhoto/craetive

Photo credit: iStockPhoto/craetive

Google entered 2011 with two major problems that threatened the company's immediate relevance and it's long-term future:

1.) The search results on Google.com were becoming increasingly ineffective because they were littered with "web spam" and articles from "content farms" (sites creating faux content to turn as many ads as possible).

2.) Social media has been replacing traditional web search for many different kinds of information gathering and Google didn't have a legitimate play in social.

The company went a long way toward addressing the second issue in July with the launch of Google+. After several high-profile social flameouts -- such as Google Wave and Google Buzz -- they've pretty much nailed it with Google+.

To be clear, we still don't know whether Google+ will be able to win over the masses, but it has become wildly popular among tech and media professionals and it is already causing Facebook to react and make changes to buffer itself against people abandoning it for Google+. To dig deeper on this topic, read my article Why Google+ is about to change the web as we know it.

As huge as social media is, the even bigger challenge for Google has been the declining potency of its search engine. In recent years, Google searches have become a lot less useful and a lot more frustrating. It has become more difficult to find stuff that you know is out there -- even stuff that you've searched for (and found) previously. Another example is pages that have posted to the web more recently. They get overpowered in the Google algorithm by older pages that have had time to accumulate more incoming links.

The big problem is SEO -- search engine optimization. A whole cottage industry has arisen around helping sites optimize their pages to get ranked as highly as possible in Google. As a result, the sites that land at the top of Google search results have become more about which sites are best optimized rather than which ones have the best and most relevant content.

Even worse, whole companies have emerged whose entire purpose is to create low-quality content that is highly-optimized for Google and loaded up with ads to turn a quick buck. These "content farms" have become big business. One of them, Demand Media -- which hates to be called a content farm and shuns the label -- is now a public company and brags about having a close partnership with Google.

I'll let you judge for yourself whether Demand Media is a content farm. Below are four articles from its flagship site, eHow. Are these helpful or useful? Would a site that aims to serve readers and not just serve ads publish these?

(We'll talk more in a moment about whether Google considers eHow a content farm.)

Recognizing the growing risks that this stuff poses to Google's relationship with users, and ultimately its business model, the company has moved aggressively in 2011 to fix the situation. It started with acontradictory blog post in January in which Google defended the quality of its search engine as "better than it has ever been in terms of relevance" while also throwing down the gauntlet on web spam (sites that "cheat their way into higher positions in search results") and content farms ("sites with shallow or low-quality content").

Then, it dropped the real bombs --  a series of major updates to its search algorithm. These have been dubbed the "Panda" or "Farmer" or "Panda Farmer" updates (don't laugh). The first one (Panda 1.0) came in February, and it obliterated search traffic to a bunch of sites, but oddly, eHow (the site most notorious for the "content farm" label) escaped unscathed.

Google eventually unleashed Panda 2.0 in April, Panda 2.1 in May, Panda 2.2 in June, and Panda 2.5 in September. According to SEO analyst Sistrix, these Panda updates eventually crushed eHow, which relied on Google search to drive most of its traffic. Despite reports of eHow's traffic dip earlier this year, Demand Media denied that it had been hurt by the Panda updates. Then, earlier this month, the company admitted eHow's traffic problem, although it tried to brush it off as "an internal technical issue." The public hasn't been fooled, as Demand's Media's stock has fallen precipitously.

So, Google apparently bagged its big game in the Panda hunt. The problem is that it took months to do it and a lot of algorithm trial-and-error and there was plenty of collateral damage done in the process. It's as if Google looked at its backyard, spotted a bunch of dandelions, and instead of taking hand trimmers and going out and clipping them, Google decided to build a highly-advanced chainsaw to deal with it. The chainsaw eventually got rid of the dandelions but it also whacked some chunks out of the hedges, put some gashes into the ground, and took out part of the back fence.

In terms of the collateral damage, TechRepublic hasn't been immune from its affects. This site has taken some bullets in the crossfire between Google and eHow. TechRepublic has a long history of publishing in-depth tips, tutorials, and best practices that have a long shelf life and that Google has always loved because they get lots of links from around the web. TechRepublic's content is the exact opposite of both web spam (we've never been great at SEO optimization) and a content farm (we focus on fewer articles and higher-quality content), and yet the Panda updates have cut in half the amount of users that Google sends to TechRepublic.

I point this out not as sour grapes or to whine about Google picking on us. TechRepublic will be just fine. We have a large base of loyal users who regularly come to our site -- especially subscribers to our popular email newsletters -- and Google may eventually figure out how to tell the difference between a content farm tip like the ones on eHow and the in-depth tutorials you get on TechRepublic.

Still, what this all comes down to is Google's faith in the algorithm. Google says that it doesn't single out sites to include or reject in Google search results. It simply builds an algorithm that systematically finds the most relevant stuff and ignores (or removes) the least relevant stuff. Google argues that this creates a fairer and more objective system, and that introducing human filtering into the system would make it biased and subjective. While that may be true, the big question is whether human intervention would make Google search more effective, and ultimately more accurate.

The problem with the algorithm (and artificial intelligence in general) is that it has no common sense or wisdom -- at least not yet. Meanwhile, the systems that Google search is increasingly competing with for information discovery -- social search and mobile apps -- use the collective wisdom of the community or targeted experts to deliver better information more quickly than Google search, in many cases.

Despite the early success of the Google+ social experiment, the Panda updates during 2011 show that Google still believes in the algorithm above all things. The company thinks that throwing more math, PhDs, and servers at any problem is the right answer. As we've seen, that approach has started to fail Google in 2011. It has had a difficult time targeting content farms and it has ended up accidentally removing a bunch of useful content in the process. The big question now is whether Google can learn from this experience and change, or if it will eventually fade into becoming a fallback mechanism that people use when they can't find the information they need from social search (asking their Twitter or Facebook friends) or a mobile app.

Also read

This was originally published on TechRepublic.

Topics: Social Enterprise, Apps, Browser, Cloud, Google

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • Are we just hoping something will go wrong at Google Today?

    Is today any different than any other day? Nope.
    Dietrich T. Schmitz *Your
    • RE: Will Google's blind faith in the algorithm doom its future?

      @Dietrich T. Schmitz * Your Linux Advocate

      Yeah, let's not talk about Duqu.
      • RE: Will Google's blind faith in the algorithm doom its future?

        Ram U
  • algorithm is not the problem cuz Bing focuses more on algorithm

    Relying on algorithm is not the problem because Bing relies more on algorithms. Bing doesn't have a full staff web spam team to manually review websites like Google does.

    I think the problem is that Google is trying to set too many rules. Also google is trying to corner the internet so they want to weed out a lot of what they call bad content (which i don't agree with them because a lot of those content farms have thought me good stuff)

    I think Google's philosophy on "whats good for users whats good everyone" is flawed!

    I am not anti democracy but there could be a very good secret out there hidden in one of those article sites but since the site doesn't have enough pagerank or has duplicated content on it, it is not being returned in search engines hence Google failing at its primary task which is "finding stuff"

    Search engine are supposed to search and find stuff. And IMO should only rank when there are too many identical results.

    Again this all goes back to Google's ranking concept as they think of Ranking as "their opinion" which should NOT be the case. Google is a search engine and it is supposed to search and return unbiased results!
  • RE: Will Google's blind faith in the algorithm doom its future?

    It's possible. It all depends on the newbies being snared and made to believe hype about things they are competely ignorant of. And for every newbie there are many thousands of others just getting out of the ignorance circle, but who still can't find ther way through the lies and clutter to the "real" content and information about these thiings, thus reinforcing for them, the inital lies and silly hype. It takes a long time for these people to wise up to the fact that anythng fed TO them is likely a lie, and that if they can't verify/clarify the information thru other channels, it's probably misinformation and worse still lies, even though they feel they're making progress because that's what they're being fed. Clouds, anything social and myriad other sources daily embed the false information until they are to mired in it that the truth appears to be the lies because they aren't doing their own homework; they are having to rely on others to do it for them as they struggle to unlearn all the bad habits as they appear finally to the up and coming users.
    Education is the only answer and it just isn't there for these people who need it, mostly because they don't know how yet to be their own researchers. There IS NO reliable information on the 'net that is easily discernible by the lied to and misled wishing to be accurately educated. For comparison, just think about spam and how pathetically little is available information that one can be sure of and most just plaini don't care, with many going out and buying a new computer when all that's wrong with theirs is it's mired down in malware.
    So, yes, it could doom their future simply via ignorance of the very people being victimized not by this outfit, but by criminals, miscreants and greed or even intentional ignorance.
  • RE: Will Google's blind faith in the algorithm doom its future?

    It's possible; but I don't expect it to happen.
  • RE: Will Google's blind faith in the algorithm doom its future?

    It might happen, and could happen, but IMO it won't.
  • RE: Will Google's blind faith in the algorithm doom its future?

    With an algorithm, Google can honestly state that they aren't "singling out" a single company. Imagine the uproar if Google decided it didn't like Demand Media, and arbitrarily lowered their rankings? Congress, tech media and everyone and their brother would be complaining about Google's "overreach" and their "aggressive filtering".
    Yea, I can hear you saying "but Demand Media was still punished (in ranking location) by Google's moves, so what difference does it make?" Lots. Lots and LOTS. One is generic, aimed at a "type of content" versus being aimed at a single company. Big frickn' difference if you ask me.
    • RE: Will Google's blind faith in the algorithm doom its future?

      @dspake@... Uhhh..where have you been the last couple months? That is EXACTLY what Google has been accused of doing and why they are under investigation. Because several sites have shown that they either gave precedence to a Google site or one of its partners, or they "punished" companies that didn't "play ball" with them, with the now infamous e-mail sent out by a Google exec that pretty much came right out and said "Well, if you don't play ball with us, we could just scrap your site from our results altogether...how would you like that?" And these accusations are coming with evidence. Like showing that one day, a certain search term results in one site being the top result, and another site being on the second page...and then the NEXT DAY after Google signs a deal with the second site, that one suddenly moves up to the top spot and the competitor moves down to like the 5th result. There is NO way that would be a natural occurrence.
  • RE: Will Google's blind faith in the algorithm doom its future?

  • Did you see the congressional hearings?

    Actually, I would say that if Google did exercise human discretion to the point that you suggest, they would have been hit by a $1 billion antitrust action by now.
    • RE: Will Google's blind faith in the algorithm doom its future?

      @tkejlboom Again....they ARE under investigation for just that. Where have you guys been lately?
  • RE: Will Google's blind faith in the algorithm doom its future?

    "...they???ve pretty much nailed it with Google+." Not sure that I have seen anything that would agree with that based on use. of my ~396 friends on FB (which I'm picky so not many) I have a total of 5 on G+. On top of that I share my Picasa web albums for sport shoots with team families (swim dive and soccer) and locked it forcing them to get gmail (and then G+ would be there as well) I had to unlock it. No one was interested in G+ it's really popular with the Geeks, but not Joe Schmoo
  • RE: Will Google's blind faith in the algorithm doom its future?

    The field of machine learning is really just beginning to come into its own with the advent of powerful computers and big data, and the sky's the limit for where it can go in the future (WATSON's Jeopardy win was a sign of where we're heading). "Asking Twitter friends" can't possibly outperform Google's massive webcrawling and indexing operation. The ultimate source of those friends' recommendations was most likely a long-ago Google search anyway. I don't think Google has anything to worry about. "Blind faith" is the realm of gullible humans. Algorithms are empirically verified constructs with the evidence to show for all to see of their effectiveness. They're the closest thing to a sure bet you're going to get. One Dutch professor even has a standing wager challenging business people to pick a quantifiable task for which data exists and he'll run it through machine learning algorithms and pay off if a human expert is able to perform the task better than the resulting algorithms. No one's bested him yet. John R. Koza has used genetic programming (in which code "evolves", mutates and breeds like biological organisms) to produce novel, patentable electronic circuits. Some of these circuits and some of the mathematical formulas for psedurandomness he derived were novel (different from existing patents) and surpassed the performance of human inventions.
  • RE: Will Google's blind faith in the algorithm doom its future?

    I love how quickly Demand Media took down the articles to which this article linked.

    So desperate to hide the fact that they have unqualified stay-at-home moms writing medical, financial and other content.

    What a joke. Panda can't hurt them enough as far as I'm concerned.
  • RE: Will Google's blind faith in the algorithm doom its future?

    This could happen, not because its an algorithm, but because they are starting to get it a little wrong, by that I mean how they weight results.

    If I use Yahoo or Bing and search for Pizza for example, I get back places that sell or make Pizza. All to often I get back the most content from Google that containes the word Pizza, which does mean content farms etc rank high up there, and that is frustrating. This is why I have stopped using Google for search, and have switched to Bing, I find it more times than not, more accurate than Google.

    I am sure Google will get better, but their market share may slide slowly until they do (which it is a little already). But their market posistion is so strong they have ample time.

    The BIG concern for Google is not so much loosing search market share (which will happen as more people use Social Media, especially Twitter and Facebook) but the fact that advertising with Google is becoming less appealing. Why advertise with Google search when I can advertise on Facebook and focus that advertising much more (I can filter based on sex, age groups all sorts). I think Facebook is stealing revenue from Google here, and it will continue to do so. This is why Google+ was launched, there is no money in search, only advertising. If Google starts to loose advertising revenue, it doesnt matter how many prefer Google search, the point is they wont have the funds to do many other things that they do.

    Google+ is their attempt to stop advertising revenue leaving Google and moving to Facebook. Though some call Google+ a success because of the numbers using it, ask youself this, who is using it? I only know of techies and journos, no one else. The average consumer out there arent even aware of Google+, but they are more than likely to be on Facebook. I cant see this changing...

    I think the issue for Google is blind faith in advertising revenue funding everything they do. If they loose advertising revenue then their business model soon comes crashing down. Social Media advertising grows, Googles advertising revenue will fall, if other search engines start becoming more popular (Bing) then again, thats more advertising revenue leaving Google.

    Its a concern as advertising is their only income stream......
  • RE: Will Google's blind faith in the algorithm doom its future?

    They've pretty much nailed it with Google+? TWICE already Google+ has seen its traffic decline by 2/3. Think about that. Not once but twice were they unable to hold on to more than 1/3 of their users. First was after launch, and the second time was after they got rid of invites. And they haven't made a dent in Facebook's lead AT ALL. (In fact, Facebook has increased by MORE than Google+ since June.)
  • RE: Will Google's blind faith in the algorithm doom its future?

    Google has serious conceptual flaws that have seeped in over the last year or two.

    - Commercial links, which are becoming the cornerstone of Web 2.0 economy, rely on duplicated content to get their message across.
    - They have inconsistent treatment of bot handling techniques that does not take real world concerns into account. Content can be blocked from bots by robots.txt, but if the same content is blocked within a page, it is treated as a black-hat attack on Google.
    - They don't provide for small business and lone web developers within their high maintenance rule set. Not everyone has a staff of thousands.
    - They don't take responsibility for quality, but instead issue edicts for what they feel is proper, expecting the world to follow or be delisted. This stifles growth and new ideas.
    - Technical search results have become littered with outdated content that is no longer applicable to modern tech.
    - General results are littered with meaningless social content and commentary.
    - Google has become heavy handed in their enforcement of AdSense policies rather than taking responsibility for ad placement on relevant sites, they have gravitated toward delisting of sites based on content.
    - Their delisting, warning and account cancellation practices have also grown heavy handed. Warnings, when issues, are vague and unclear. Account cancellations at Adsense also affect other Google services such as the affiliate network and youtube. The end result is that what should result in Google just excluding a site from ad placement, ends up being the destruction of livelihood.

    Google would be better served by just returning random results from the database matching the queries. The result would be equitable display of all content, rather than Google-optimized content.

    Providing channels within search would also be useful. When I'm shopping, I want to see all the sites that offer a product. I don't want to see TV news stories and twitter feeds. And I don't want small companies and thin shopping portals to be excluded - they provide the best deals.

    When I'm doing tech research, I only want tech results.

    When I'm search for social media or pop culture, then show me that.

    I'd be surprised if Google wasn't replaced by a competitor in the next five years.