Rupert Goodwins' Diary

Friday 07/09/2001

While trying out the WiseNut search engine -- which looks as if it may one day be a serious competition for Google -- I run the industry standard benchmark, Ego 1.0. You can do it, too: just type in your own name and see what it comes up with. In my case WingNut managed around 80 per cent more than Google, including some distinctly dodgy hits: I'm not sure I want a story I wrote about mobile phones and cell proteins propping up an esoteric bookshop which is hot on astrology and primal channelling.

The best one, though, was a Taiwanese Web site that used a clip of an interview I did for the World Service to teach English. The thought that even as you read this I'm annoying a school kid in the Far East is curiously satisfying... although on inspection, it turns out they chose that clip for its industrial density of clichés per cubic inch.

But the real discovery was that neither Google nor WingNut nor, as far as I know, any other search engine has much by way of post processing. ZDNet news goes to a myriad places around the world, many of which do some tweaking, so looking for one thing is liable to bring up hundreds of near-identical results. And what can you do to filter these? Go through them by hand, using whatever tools the search engine offers for excluding unwanted results? Fine, but hard work and hardly useful if you want to do it very often. Or you can learn Perl and write obtuse scripts to grab results and parse them to death -- the choice of nerds, but no good to the rest of mankind.

So, given that we're drowning in information, where are the easy to use, graphical tools to categorise and handle it? I just want to point a program at a Web site, tell it what to send and then let it munch through the stuff that comes back, extracting the bits I've told it I want and being smart about deduplication, categorisation and presentation.

Anyone know of such a thing?

To have your say online click on TalkBack and go to the ZDNet forums.