Quocirca's Straight Talking: The search for intelligent web info

The truth is out there... or is it?

Is there a way to make the information we find on the web more reliable? It won't be easy but Quocirca's Clive Longbottom has a few ideas.

I was recently sitting listening to one of the large search engine companies - Yahoo! Europe - talk about their ongoing plans for global domination, when it struck me that the shrinking of the world through the use of the internet is reproducing a lot of the issues that we have had in the past. If we look at the new 'global village', we can compare a lot of what we see against what we had in the villages of old.

Firstly, we have the rumour mill. In the past this meant the village gossip would hear something (generally second or third hand) and would embellish it and pass it on. This would be received as the truth until it could be proved otherwise by a large enough group of villagers.

Secondly, we have the conspiracy theorist who would spread the idea that anything that went wrong would be the fault of others, who would have caused it to go wrong for their own ends. This is harder to disprove but, again, provided enough sane voices were raised, the conspiracy would eventually go away.

Then there's the soothsayer who said whatever went wrong was due to bad auguries - and that everyone was doomed due to their bad behaviour. This person may cause some visceral feelings in the rest of the village but would not generally cause too much in the way of major problems.

We also have the various groupings of like-minded people who get together in the working men's clubs, the Women's Institute halls, the religious meeting places and so on, and feed off their own beliefs to the exclusion of outside influence.

Finally, we have the village idiot - a useful person that the rest of the villagers with all their own prejudices and misconceptions could look down on and blame for everything.

Each village was pretty insular. It was the nomads and travellers who carried the news around - and they often had their own agendas and bigotries that coloured what they passed on.

So here we are today with the new global village - 6.5 billion people with disparate beliefs, perceptions and agendas. The vast majority are still disconnected from the main fabric of the village - sort of like the farm workers in the villages of the past who would get news handed down by passing people who had heard it from someone talking to someone else in the village market.

For those who are part of the village proper, surely we now have a cognoscenti, a group of people who are providing The Truth, the information that is needed by the rest to help make their world a better place, for them to make more informed decisions. Don't we?

Well, no. The few cognoscenti are like the old prophets, generally unrecognised during their own lifetime. What we have are 'communities of interest', highly specialised and focused groups of like-minded people with their own agendas.

Although some of these are small, there are many large groups whose membership can be measured in the hundreds of thousands or the millions. And, whereas in the physical village, one person against 10 could actually become the voice of sanity and disprove a point held by the majority, these communities are far more dangerous.

As an example of a modern 'community of interest' consider Slashdot. Try leaving a post on Slashdot that says Linux on the desktop is a small blip in modern commerce, and see what result you get. The massive backlash from the overwhelming majority beats any dissenting voice to the ground.

It's similar with the conspiracies. Post information on the web that says that Princess Di simply died due to bad driving and a bunch of idiotic paparazzi, and you'll get hit with gigabytes of so-called 'proof' that it's all down to Prince Philip, MI5 and a laser gun.

Ditto with the religious groups. Whereas in the villages of old, breakaway groups tended to remain small and would peter out once it became apparent that the new leader was a nutter with only his own interests at heart, the global village enables thousands to millions of new people to follow the nutter - and to propagate information that would threaten anyone who dares to gainsay him.

These days we are also giving more soapboxes to those who want to stand and rant. Blogging is on the increase - at least the number of people who write blogs is growing. Our research shows that blog readership is still miniscule, and is moving more towards community-of-interest style usage. It seems the amount of 'good' information is rapidly being hidden under the mountain of 'dubious' information.

Take the search engine players. They are not in the market to censor the content of the internet (apart from when they are trying to crack the Chinese market); their job is to make the content easier for people to find, whether the information is right or wrong. However, the lack of verifiable content on the web means search engines now have to look at other ways of presenting this information.

Yahoo!'s idea is to use a ranking system - content will be ranked according to a democracy whereby information that others have found useful will be higher ranked; those people who provide consistently well-ranked content will be seen as 'authorities' and their output will be considered of high value.

This is a lot better than what we have at the moment but has its own issues too. Let us assume that an article is posted by Dan Brown, the author of The Da Vinci Code, stating that Opus Dei and the Vatican are in some conspiracy. We can pretty much bet that this would get ranked as a useful piece of information by an authority - even if it is based on dubious assumptions.

How about the thoughts of George Galloway? He would probably get strong backing from elements of the community but does it make him right?

In this ranking system, the likes of the gossip mongers would grow in strength and the numbers of village idiots would multiply exponentially.

Yahoo!'s view is that we should let the natural cynicism and intelligence of the average user hold sway - and that these users would not allow themselves to be undermined by the content of the web.

My view is far more bleak. Scott Adams said: "Never underestimate the stupidity of the general public." (I thought it pre-dated him but, hey, it's on the web that he said it, so it must be right.) In this case, we must never underestimate the public's stupidity. If information is there, plenty of people will believe it. If plenty of people believe it, it becomes the truth. If enough people believe it as the truth, there'll be enough people to fight for it. And so the fighting nature of mankind is perpetuated.

Can we do anything about it? Is censorship the answer? I don't think so - who will look after the censors to ensure that free speech is allowed? No, what we need is a means of better interlinking information sources, of breaking through the walls of the communities of interest, the secret clubs, the ivory towers of the bigots and the manipulators.

We need to look at how we can 'peer manage' information - a system whereby a ranking from someone already seen as an authority would be better marked than one from a non-authority.

Yahoo! is looking at how meta tags can be utilised in this way. It has a lot of information on Yahoo! users which could be used for advanced profiling. However, many of us have two or more online personas. Our work usage of Yahoo! might be completely different to our home usage, for instance, and this dichotomy may skew our profile as to how authoritative we actually are in specific circumstances.

It will never be perfect but we all have a responsibility to stop the reliability of online information from getting worse. It's far better to be one of the new cognoscenti than one of the inbred global village idiots.