ie8 fix
Click Here

Linux and Open Source

Steven J. Vaughan-Nichols & Paula Rooney

Should search engines pay tribute to content?

By | November 16, 2009, 2:43pm PST

Summary: There’s some basic math at work here. The smaller your circulation base, the more specialized it is, the better off you’ll do with either a paid model or a registration model.

Tom Foremski, one of the good guys here at ZDNet, is out with a piece suggesting that Google fork over whatever Rupert Murdoch wants in order to keep indexing Fox News.

His argument is that losing access to regularly-updated content would be a big hole in Google’s business model, which is based on making everything out there available.

I have two problems with that:

  1. If one publisher can force Google to pay for a link, so can every other publisher. Blackmail never ends.
  2. Publishers have tried this before and failed. Including Murdoch.

I think the first point is more important. Foremski argues that newspapers get little traffic from Google. This is true, mainly because, as he notes, most still haven’t got a Clue when it comes to the Internet.

Google has already taken their wire service traffic away, signing side deals with major wire services like AP, posting those stories on Google pages, paying back the ad revenue. The only newspaper content left is local or beat-specific. Most traffic to those stories comes from the local area.

So if the traffic flow is modest, why should Google be paying? Just to protect its reputation?

There is no need to argue the point. We can run an experiment.

Let Bing or Yahoo pay Murdoch, and let Murdoch put robots.txt files on all his properties, keeping them away from Google’s crawlers. See what happens. If there’s real market share to be gained here, Google’s competitors will be happy to buy it.

On to the second point. Both The Wall Street Journal and The New York Times have tried the paid model. The Journal maintains it, but offers links to the full-text of its stories, through Google. The Times gave up its Times Select program as a money-loser.

Yes, The Journal began its program before Google bought it. But I also remember a time, about a decade ago, when Fox site managers were transferring every link deep into their site to the home page. It was maddening. They stopped.

There’s some basic math at work here. The smaller your circulation base, the more specialized it is, the better off you’ll do with either a paid model or a registration model.

Lots and lots of journals allow only access to abstracts if you’re not a registered user. The New England Journal of Medicine is an example. And there are many publications only available to paying customers, who are notified of updates via e-mail.

The problem is that if you want a mass audience on the Internet, you have to make yourself available to a mass audience. Newspapers are mass circulation publications. Throwing up registration windows or pay walls hasn’t worked for them.

But, again, Murdoch (and every other publisher) is perfectly free to try. Just write a simple robots.txt file forbidding indexing. Poof, you’re invisible to the search engines’ spiders.

There’s no real controversy here, IMHO. Publishers are free to conduct what experiments they want with Google, either seeking to tweak it to get more audience, or block it to access a smaller, elite audience.

History says Murdoch is barking at the Moon. But he likes to write his own history. Let him try.

Kick off your day with ZDNet's daily e-mail newsletter. It's the freshest tech news and opinion, served hot. Get it.

Topics

Dana Blankenhorn has been a business journalist for 30 years, a tech freelancer since 1983.

Disclosure

Dana Blankenhorn

Dana Blankenhorn has been a journalist, writer and part-time futurist for over 30 years.

At the present moment I run only a personal blog in addition to my ZDNet open source blog.

DanaBlankenhorn.Com has the subtitle The War Against Oil. In the past I have used it to write about political history, e-commerce, personal matters, some ideas related to open source, and The World of Always On, which is the idea of using sensors, motes and RFID to turn WiFi links into platforms for applications which live in the air.

My IRA account at Schwab holds a few tech shares, most notably some Intel and Applied Materials, but there are no open source companies in it. I don’t even own any CBS stock.

Biography

Dana Blankenhorn

Dana Blankenhorn has been a business journalist for nearly 25 years and has covered the online world professionally since 1985. He founded the Interactive Age Daily for CMP Media, and has written for the Chicago Tribune, Advertising Age's "NetMarketing" supplement, and dozens of other publications over the years.

Related Discussions on TechRepublic

Did you know you can take part in these discussions with your ZDNet membership?
30
Comments

Join the conversation!

Just In

You seem to forget one thing.
hkommedal 18th Nov 2009
The traditional payment model for newspapers is like this.

Advertizing pays for the content.
Subscribtion pays for the distibution.
0 Votes
+ -
Why?
SystemVoid 16th Nov 2009
Why should Google have to pay "royalties" for
showing people their way to content on the
internet? Should Digg.com and Delicious.com
have to do the same thing? If I put a link (or
links) on my website, to say Foxnews.com, does
that mean I should have to pay Murdoch
royalties for doing so?

Perhaps we should just get rid of hyperlinks
all together, but only the ones that send users
to any site but the domain they're already on.
Problem solved. That should make Murdoch happy.

What's really amusing about all of this crap
Murdoch is yapping about, is the shear billions
Google makes from people paying THEM to send
traffic to their websites. Maybe Google isn't
for everyone, as Murdoch would like to think.
But for the remaining 99.9999% of the internet
universe, it really does.

Murdoch complains. Google yawns. Techies laugh.
And the rest of world doesn't give a damn. The
End.
0 Votes
+ -
Until, of course, when all journalists are out of business. This should happen soon enough because content providers cannot continue to pay for staff to write "free" information just so that Google can make billions on search.

As much as RMS would like to think that the "information wants to be free", you HAVE to pay someone, and pay them rather well, for the research, writing skill, and reputation to create content people want to read. Google does not put one nickle into news (or most any other) content gathering, and so is just a grifter in this scenario.

0 Votes
+ -
You seem to forget one thing.
hkommedal 18th Nov 2009
The traditional payment model for newspapers is like this.

Advertizing pays for the content.
Subscribtion pays for the distibution.
other sites, let him pull the links. Seems like he has
more to lose than Google. It would be different if there
were not other websites with the same information.
0 Votes
+ -
Agreed
Michael Kelly 17th Nov 2009
If the yellow pages printed a full page listing for your business for free, who in their right mind would object?
0 Votes
+ -
they are floundering in a greed based society - good luck ya fools
0 Votes
+ -
Not really
DanaBlankenhorn 17th Nov 2009
They're changing, evolving, growing more specialized.

Take Atlanta, for instance. We have papers on every subject the Atlanta Journal-Constitution covers, and papers in every suburb (plus many neighborhoods). Most are free.

Then there are the news-oriented Web sites.

The business is evolving, and those who don't evolve are going to die. That's capitalism.
0 Votes
+ -
Evolving: Get Writers to Work for Free
Too Old For IT 17th Nov 2009
... of course, I'm not sure how they would be able to keep house, home, and body together while writing for free for an unappreciative society.
0 Votes
+ -
let him try?
ljenux-23043766007667558234416105604265 16th Nov 2009
why not jailing him instead and giving media back to journalists, so that people have independant media instead?
0 Votes
+ -
The nations loss when Newspapers fail.
No_Ax_to_Grind 17th Nov 2009
The truth here is Newspapers need a revenue stream to stay in business. That has traditionally came from exchanging well written news articles for advertizing dollars. As Google is collecting the ad revenue the Newspapers are left wanting. The end result of course is the decline of newspapers and in some cases their complete disappearance. Sadly, that means readers are stuck with opinionated "blogs" like we see here at ZDNet. Hey, blog are nice and all, but they are not professionally written news articles and yes, it shows.
0 Votes
+ -
You mean like Fox News?
Michael Kelly 17th Nov 2009
And the New York Post?

Sorry, but if Google were to grow a heart and give back to real journalism, giving anything to a News Corp. subsidiary would be counterproductive.
0 Votes
+ -
Not a universal truth
Real World 17th Nov 2009
I find it hard to believe that you're claiming that ALL blog posts are of lesser quality than ALL news articles. So-called professional journalists have been blurring the line between objective fact and subjective opinion for as long as I can remember, and determining the truth has pretty much always been left as an exercise for the reader.

And I'm not even going to start on news agencies pushing their political agendas...
0 Votes
+ -
No such thing as a universal truth
No_Ax_to_Grind 17th Nov 2009
But when talking in generalities it is true, blogs pretty much suck when it comes to the facts. I look at ZDnet as a perfect example. It is anything but a news site today and more closely resembles Twitter or Facebook with opinions flying around in place of facts.
0 Votes
+ -
My point was
Real World 17th Nov 2009
it isn't any more or less reputable than a traditional news site. As with anything, buyer beware.
0 Votes
+ -
I understood your point, and disagree.
No_Ax_to_Grind 17th Nov 2009
Blogs suck as a news source, period, the end.
0 Votes
+ -
ok
Real World 17th Nov 2009
Blogs suck. So do newspapers. TV, web sites, any other form of media you can imagine.

Wow, this is a worthwhile discussion...
0 Votes
+ -
Completely disagree on this one.
terry flores 17th Nov 2009
Newspapers are just as prone to printing opinionated garbage, political bias, and advertising disguised as news and any blog out there. Journalistic integrity in print is a myth, just as it is on TV and on the Web.

Newspapers need a competitive edge to get that revenue stream; they are not "entitled" to stay in business if they can't compete. And "well-written" is in the eye of the beholder, who ultimately decides whether it's worth the money or not.

It remains to be seen if newspapers can reinvent themselves from a model that currently depends on using massive amounts of "wire" news that is easily available from multiple sources, or being specialized news sources for local events are not reported elsewhere. If they can't figure out how to make a living in a niche, then they are doomed.
0 Votes
+ -
Disagreeing doesn't change it.
No_Ax_to_Grind 17th Nov 2009
Blogs suck as a news source, period.
0 Votes
+ -
I've been a professional journalist 31 years
DanaBlankenhorn 17th Nov 2009
The big difference between using this blog format and the old inverted pyramid is you can dump on me, and I can respond as here.

There are also different rules for blogging, which I've discussed on my personal blog.

Doesn't mean it's not journalism. It means it's what the market now wants. That's always been my definition of journalism -- what the market wants.

Don't you like the market? I do.
That explains the poor quality of you blogs/journalism.
0 Votes
+ -
Journalism
bhartman36 17th Nov 2009
Doesn't mean it's not journalism. It means it's what the market now wants. That's always been my definition of journalism -- what the market wants.

In practical terms, journalism might be about what the market wants, but journalism is supposed to be about keeping the citizenry informed. Journalism should be about important topics, not just "what the market wants". The market might want TMZ-style information, but does that even have a passing resemblance to real journalism?

I recently swore off CNET, and part of the reason was they've become unreliable in their reviews. A recent "prizefight" pit Windows 7 against Snow Leopard, and the conclusions of that were enough to convince me that CNET has lost all sense of objectivity. Catering to Apple fanboys might be what the market wants, but when it conflicts with objective reality, actual journalism dictates that you call it as it is.

In regards to the Murdoch question, I don't see any reason why Google should pay for content. If you don't want Google crawling your content, just put up a robots.txt file. It's not rocket science. Just watch what happens to your revenues when you do that, though. The whole point of being listed in Google results is that people get to your content, and advertisers benefit from that.

The paid subscription model does not work on the Web. Just ask the people at the NY Times. Even Amazon is struggling, as far as I can tell, to justify selling subscriptions to Kindle newspapers.

You can't put the toothpaste back in the tube, but hey, good luck with that, Murdoch! happy
0 Votes
+ -
Google get's revenues from sponsored links (using the Pay Per Click model) or paying for placement. Sites that have advertising on them make their own revenues from the display ads or links. You appear to not understand how Internet based advertising works and what Google (or other search engines) do to generate advertising revenues. There are several sites that make their revenues from advertising that are indexed by Google (and the other search engines).

The problems that the newspapers are facing (paper based) is that it costs more to produce the physical content (paper, ink, electricity, etc.) than it does to publish it in an electronic form (on-line, on a Kindle or similar device, etc.). These costs are covered by subscriptions, advertising and the (now disappearing) coin operated news paper dispensing racks (or retail establishments, bookstores, etc.). All of these also have a cost.

Blogs (such as ZDNet content) are covered via PPC/banner ads. Unfortunately, blog content tends to be rather unresearched opinion pieces similar to that which Hunter S. Thompson used to write (but with lower quality than HST). The problems also stem from the culture of "free" that has come from the use Web based sources with no revenue models or access controls/subscription models.

A robots.txt file (as per the standard) looks like:

User-agent: *
Disallow: /search
Disallow: /groups
Disallow: /images
Disallow: /catalogs
Disallow: /catalogues
Disallow: /news
Allow: /news/directory
Disallow: /nwshp
Disallow: /setnewsprefs?
Disallow: /index.html?
Disallow: /?
Disallow: /addurl/image?
Disallow: /pagead/
Disallow: /relpage/
Disallow: /relcontent
Disallow: /imgres
Disallow: /imglanding
Disallow: /keyword/
Disallow: /u/
Disallow: /univ/
Disallow: /cobrand
Disallow: /custom
Disallow: /advanced_group_search
Disallow: /googlesite
Disallow: /preferences
Disallow: /setprefs
Disallow: /swr
Disallow: /url
Disallow: /default
Disallow: /m?
Disallow: /m/?
Disallow: /m/ig
Disallow: /m/images?
Disallow: /m/lcb
Disallow: /m/news?
Disallow: /m/news/i?
Disallow: /m/setnewsprefs?
Disallow: /m/search?
Disallow: /m/swmloptin?
Disallow: /m/trends
Disallow: /wml?
Disallow: /wml/?
Disallow: /wml/search?
Disallow: /xhtml?
Disallow: /xhtml/?
Disallow: /xhtml/search?
Disallow: /xml?
Disallow: /imode?
Disallow: /imode/?
Disallow: /imode/search?
Disallow: /jsky?
Disallow: /jsky/?
Disallow: /jsky/search?
Disallow: /pda?
Disallow: /pda/?
Disallow: /pda/search?
Disallow: /sprint_xhtml
Disallow: /sprint_wml
Disallow: /pqa
Disallow: /palm
Disallow: /gwt/
Disallow: /purchases
Disallow: /hws
Disallow: /bsd?
Disallow: /linux?
Disallow: /mac?
Disallow: /microsoft?
Disallow: /unclesam?
Disallow: /answers/search?q=
Disallow: /local?
Disallow: /local_url
Disallow: /froogle?
Disallow: /products?
Disallow: /products/
Disallow: /froogle_
Disallow: /product_
Disallow: /products_
Disallow: /print
Disallow: /books
Disallow: /bkshp?q=
Allow: /booksrightsholders
Disallow: /patents?
Disallow: /patents/
Allow: /patents/about
Disallow: /scholar?
Disallow: /complete
Disallow: /sponsoredlinks
Disallow: /videosearch?
Disallow: /videopreview?
Disallow: /videoprograminfo?
Disallow: /maps?
Disallow: /mapstt?
Disallow: /mapslt?
Disallow: /maps/stk/
Disallow: /maps/br?
Disallow: /mapabcpoi?
Disallow: /maphp?
Disallow: /places/
Disallow: /maps/place
Disallow: /help/maps/streetview/partners/welcome/
Disallow: /lochp?
Disallow: /center
Disallow: /ie?
Disallow: /sms/demo?
Disallow: /katrina?
Disallow: /blogsearch?
Disallow: /blogsearch/
Disallow: /blogsearch_feeds
Disallow: /advanced_blog_search
Disallow: /reader/
Disallow: /uds/
Disallow: /chart?
Disallow: /transit?
Disallow: /mbd?
Disallow: /extern_js/
Disallow: /calendar/feeds/
Disallow: /calendar/ical/
Disallow: /cl2/feeds/
Disallow: /cl2/ical/
Disallow: /coop/directory
Disallow: /coop/manage
Disallow: /trends?
Disallow: /trends/music?
Disallow: /notebook/search?
Disallow: /musica
Disallow: /musicad
Disallow: /musicas
Disallow: /musicl
Disallow: /musics
Disallow: /musicsearch
Disallow: /musicsp
Disallow: /musiclp
Disallow: /browsersync
Disallow: /call
Disallow: /archivesearch?
Disallow: /archivesearch/url
Disallow: /archivesearch/advanced_search
Disallow: /base/search?
Disallow: /base/reportbadoffer
Disallow: /base/s2
Disallow: /urchin_test/
Disallow: /movies?
Disallow: /codesearch?
Disallow: /codesearch/feeds/search?
Disallow: /wapsearch?
Disallow: /safebrowsing
Allow: /safebrowsing/diagnostic
Disallow: /reviews/search?
Disallow: /orkut/albums
Disallow: /jsapi
Disallow: /views?
Disallow: /c/
Disallow: /cbk
Disallow: /recharge/dashboard/car
Disallow: /recharge/dashboard/static/
Disallow: /translate_a/
Disallow: /translate_c
Disallow: /translate_f
Disallow: /translate_static/
Disallow: /translate_suggestion
Disallow: /profiles/me
Allow: /profiles
Disallow: /s2/profiles/me
Allow: /s2/profiles
Disallow: /s2
Disallow: /transconsole/portal/
Disallow: /gcc/
Disallow: /aclk
Disallow: /cse?
Disallow: /cse/panel
Disallow: /cse/manage
Disallow: /tbproxy/
Disallow: /comparisonads/
Disallow: /imesync/
Disallow: /shenghuo/search?
Disallow: /support/forum/search?
Disallow: /reviews/polls/
Disallow: /hosted/images/
Disallow: /hosted/life/
Disallow: /ppob/?
Disallow: /ppob?
Disallow: /ig/add?
Disallow: /adwordsresellers
Disallow: /accounts/o8
Allow: /accounts/o8/id
Disallow: /topicsearch?q=
Disallow: /xfx7/
Disallow: /squared/api
Disallow: /squared/search
Disallow: /squared/table
Disallow: /toolkit/
Allow: /toolkit/*.html
Disallow: /qnasearch?
Disallow: /errors/
Disallow: /app/updates
Disallow: /sidewiki/entry/
Disallow: /quality_form?
Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
Sitemap: http://www.google.com/hostednews/sitemap_index.xml
Sitemap: http://www.google.com/ventures/sitemap_ventures.xml
Sitemap: http://www.google.com/sitemaps_webmasters.xml
Sitemap: http://www.gstatic.com/trends/websites/sitemaps/sitemapindex.xml
Sitemap: http://www.gstatic.com/dictionary/static/sitemaps/sitemap_index.xml
0 Votes
+ -
Wow I agree that opinion...so Should search engines pay tribute to content
http://www.canada-radio.co.tv/
http://www.canada-radio.co.tv/
http://www.canada-radio.co.tv/
0 Votes
+ -
As a BOFH title says...
benitodarder 17th Nov 2009
"When non IT-people makes IT-decisions", I've the feeling Murdoch and friends don't understand how and what means internet.

0 Votes
+ -
Conversely ...
Too Old For IT 17th Nov 2009
Conversely, IT-People do not understand what it takes to publish newspapers, music, books ... anything.

The just assume that they can steal another person's hard work because they are cheap and want everything handed to them for free.

The internet, in Objectivist terms, is the ultimate "looter society".
0 Votes
+ -
All Murdoch has to do is employ a robots.txt solution. Anyone who's getting crawled by Google is allowing them to do so. It's not hard to prevent it.

And what it takes to publish successfully on the Internet is very well-understood. Most outfits that try to charge for content directly (rather than through advertising) fail. It's not rocket science. If you want to profit, you get advertisers to pay for space. Just ask the people at the New York Times how well charging for content works out. The only thing they've been able to sell so far, despite repeated attempts, has been access to the crossword (which is the one thing you can't get anywhere else in that kind of quality).

Murdoch has an inflated sense of his empire's importance. He's in a similar scuffle with Amazon over the Wall Street Journal on the Kindle. He raised the price of a WSJ subscription on the Kindle to $15 from $10, and watched the subscription rate drop like a rock (along with the review ratings).

Bear in mind: I'm not against people getting paid for their work. Quality content deserves to get compensated adequately. But Murdoch is competing in a realm now where people have long ago realized that you can't make money by paid subscriptions. That doesn't mean that you can't make money at all, but it means that a subscription model's not going to work. Holding your fingers in your ears and screaming, "La la la! I can't hear you!!!", as Murdoch is doing right now, doesn't change that. In Objectivist terms, "A is A".
0 Votes
+ -
Google News feeds readers to the newspapers.
peter_erskine@... 17th Nov 2009
If you go to Google and click on News, you get the latest news with links to the newspapers. In the UK, the Guardian, Telegraph, and Times are often there. So they get millions of readers, from Google, with zero cost or effort on their part.
A publisher would indeed be stupid to try and charge or block Google!
0 Votes
+ -
and for some to suggest google is taking away ad revenue that would otherwise go to these newspapers is just nonsense.
The news corporations need to secure their own advertising for their websites, and should be thankful search engines provide links to their sites for free. There is no other medium that basically provides free advertising like search engine results.

The reality is that the newspaper industry is going to have to downsize, less people purchase newspapers nowadays, more checkout their websites and other sources instead, and if the advertising revenue is less because of this evolution of behaviour, they just have to accept it and change. Yes it most likely means less traditional reporters will be employed, but it's not like they're going to go extinct, there will always be a market for professionally produced news.
Murdoch just can't wrap his head around the idea of his traditional empire shrinking.
0 Votes
+ -
Search engines are parasites anyway...
Marty R. Milette 17th Nov 2009
Search engines make a profit by indexing content that they have not invested in creating.

They consume the bandwidth from the provider of the content (and the entire Internet) yet offer nothing in return except MAYBE some visitors and certainly no share of the hard cash THEY are making out of the deal.

I've had numerous web sites knocked off line by overly agressive crawlers -- effectively a denial of service attack.

Would be nice to get even a small share of the money Google and others are making from my content...
0 Votes
+ -
RE: Should search engines pay tribute to content?
brighteyes459@... 18th Nov 2009
It is not Google's responsibility to protect a business
web site, they are a search engine, not the internet
police. I have been to sites that had free areas and
member only areas, either I pay or have to look else
where.

These sites/business need to take responsibility for
themselves and stop trying to put the blame on Google or
any other search engine for their content being seen.

Join the conversation!

Formatting +
BB Codes - Note: HTML is not supported in forums
  • [b] Bold [/b]
  • [i] Italic [/i]
  • [u] Underline [/u]
  • [s] Strikethrough [/s]
  • [q] "Quote" [/q]
  • [ol][*] 1. Ordered List [/ol]
  • [ul][*] · Unordered List [/ul]
  • [pre] Preformat [/pre]
  • [quote] "Blockquote" [/quote]
ie8 fix

The best of ZDNet, delivered

ZDNet Newsletters

Get the best of ZDNet delivered straight to your inbox

Facebook Activity

White Papers, Webcasts, & Resources
ie8 fix