Blink - Malcolm Gladwell on our mysterious decisions

Why does "open source intelligence" work? Isn't it counterintuitive that lower-grade information, as publicly available information should be, would lead to as good as or better analysis as that based on high-quality intel? Malcolm Gladwell, author of "Blink," addressed this issue briefly in a speech to Hamline University Law School (click here to download MP3).

Take for example the wildly off CIA estimates of Soviet military power. Journalists and academics came to much better estimates than did the professionals, in spite of - or precisely because, as Gladwell argues - limited access to information. Too much information seems to pollute our judgement, Gladwell says.

The gold standard of this argument is a study that found that emergency room doctors' diagnoses of heart failure could be dramatically improved by limiting the information inputs to four critical data points. And in New York, a counterterrorism chief reduced from 42 to 5 the number of criteria officers could use in deciding whom to intercept and question. The number of "hits" went from 7% to 14% and the amount of material intercepted tripled.

The intelligence study suggests that it's not even so much what datapoints you retain, so long as you strongly filter out a healthy amount of the datastream.

So how could this relate to IT? Obviously, government IT pros are faced with a huge amount of data on an ongoing basis - customer complaints, vendor sales information, a flow of news and blogs and opinion. Gladwell's insights suggest it would be best to strongly limit the amount of information you take in in making IT decisions, but this limitation idea is just one aspect of his insights.

Equally important are the idea that we typically make decisions instantly and that all of our careful deliberations are basically used to backup our intuitive choice, that we need to be aware that we do this, that the intelligence of intuition can easily be corrupted by environment (as an experienced IT pro, your inuition might tell you to use open source but if you're in a Microsoft shop you'll work out a rationale that makes MS the best choice), and that extraneous information can lead to bad decisions.

An eye-opening (or closing) lecture and book.