The burden of the spiderbots and net neutrality

The burden of the spiderbots and net neutrality

Summary: I would love to know how much Internet bandwidth is used by the swarms of spiderbots?  Because if bandwidth costs are going to go up, as the telco/cable last mile owners charge third parties for bandwidth, then the spiderbots might get banned.

SHARE:
TOPICS: Browser
4

I would love to know how much Internet bandwidth is used by the swarms of spiderbots?  Because if bandwidth costs are going to go up, as the telco/cable last mile owners charge third parties for bandwidth, then the spiderbots might get banned.

Spiderbots eat up a huge amount of bandwidth, and if bandwidth gets more expensive, the spiderbots are going to suffer. For example, I get 5 per cent of my traffic from more than 18 spiderbots. They use up about one-third of my bandwidth.

That's a key reason why Google, Yahoo and others, are pushing for net neutrality--equal access to bandwidth--at least the last mile pipe to the home, the most important pipe. If companies are going to have to pay extra to the telcos or cable companies for bandwidth to reach their users, they might not be so pleased to be paying for the bandwidth of the spiderbots.

I'm fortunate that more than 92 per cent of my readers come directly through bookmarks or RSS, so they know where I live. Many other sites depend on 30 per cent to 70 plus per cent of their traffic from the search engines.

And they spend a lot of money to optimize their sites to attract more spiderbots. But this is not all quality traffic, it is mostly fly-by-night web surfers. Web sites should optimize for their readers, not the spiderbots. Let the search engines optimize themselves, that's their job.

When audience numbers stabilize for a web site, and very few new readers come in from the search sites--yet the spiderbots suck up one third of the bandwidth--then things will change. More and more web sites will be posting a Robot.txt file that tells the spiderbots to go away. They will change because the overall visitor experience is slowed down by the bandwidth hungry packs of spiderbots.

We used to have estimates of how much bandwidth is used by email, by SPAM, etc, how about spiderbots? I would love to know: how much Internet bandwidth is used up by the legions of spiderbots, in their constant search to find and copy new content?

Topic: Browser

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

4 comments
Log in or register to join the discussion
  • Interesting perspective

    Very interesting point you make on the spiderbots. Just another example of why tiered service really just makes sense. If I use my internet access to download HD video all day long, and you use yours to check your email once a week, it just makes sense for you to have to pay less. Any other way means you're paying for my internet service.
    Hands Off The Internet
  • Outstanding question!

    Your very insightful analysis exposes that the search engines are not paying that much for their bandwidth.

    Google et al say they pay a lot for bandwidth, but nowhere near what use.

    Why net neutrality is essentially special interest legislation and corporate welfare for dotcom billionaires, is that those entities that use the Interent the most, search engines, want to pay the least. The search engines have 80-90% gross profit margins and don't want their costs to go up becuase their profits will come down.

    Its more profitable for broadband providers and users to subsidize the search engines use of bandwidth.

    You are on to something!
    scleland@...
    • Uh, dude.

      Unless your blog is hosted on a home computer, spiderbots don't touch your last mile, they're already on the tiered Internet interrogating the server where your blog lives.

      That's the way most people do it, at any rate.
      richardbennett
  • Once Veri$on hijacks the net from everyone, would it realy matter?

    You may not have known this unless your were a Veri$on DSL customer, but they have a search engine as well. All it does is take results from other search engines including Google. Wonder how much precious Veri$on bandiwidth that soaks up?

    Not that they're huring for bandwith because of their customers usage, their [url=http://blogs.zdnet.com/Ratcliffe/index.php?p=155]fine print in their "Usage Agreement" along with an example of what happened to one customer who "violated" it[/url] will almost guarantee that Veri$son will have ample bandwith, along with minimal VoIP competition, for their lame search engine, sponsored advertisements, and partners'/co-conspirators' ideologies. At that time, people could care less about QoS or access speeds since the net will be nothing more than a giant propaganda botnet vomiting Veri$on/Government Borg-think on our hard drives.

    I hate to think what the net would be like if [url=http://news.zdnet.com/2100-1035_22-6091853.html]Cingular was an ISP instead of a wireless provider.[/url]
    Mr. Roboto