Lies, damn lies and statistics

During my morning news scan, I came across a variety of things coming out of industry events that appear to offer grand conclusions from a fairly small, focused studies. It gives me an opportunity to point out the limitations and uses of small studies across the industry.

Although some of these are based upon wonderful studies, the publication of these survey results gives me the opportunity to focus on a broader topic -- how small scale surveys can be used and abused by industry players.

Big conclusions, small studies

When I see broad, sweeping statements made based upon a study with a small, focused base of resondents, I become concerned that the study may not be broadly representative and, thus, may not be really useful to a broad audience. They can, on the other hand, be very persuasive to those who don't take the time to really determine the size of the sample, who was in that sample, how that sample was generated, how questions were presented to members of the sample and then how that raw data was cleaned and analyzed.

Suppliers often seek out studies that support their messages and their views. There's nothing wrong with that. When they, however, find or fund limited studies and then present them as having broad usefulness or as describing the thinking of a worldwide, broad audience, issues start to arise.

It's up to the audience to look closely at the results to determine how applicable they are to the audience's needs.

Here are some quick thoughts about short studies:

• While a small to medium size of IT professionals is a large enough sample to obtain some useful trending information, the sample isn't large enough to contain representatives from many geographical areas, vertical markets or different company sizes. So, it is unlikely to be indicative of the responses that might be obtained in a larger, multinational survey.
• Are the respondents actually decision makers within their organization?  If so, the predictions made based upon the results of the study might only be useful to represent the thoughts of a narrow segment of the market.
• Small studies can be very useful for getting a snapshot of opinions and plans, but unless great care is taken to make sure that respondents represent the broadest possible audience, the results can not be used to support broad sweeping statements. A study that includes, for example, 30 well chosen developers using a specific tool or platform may actually be a large enough sample to determine the views of a broad population of developers using the same tool or platform.  It wouldn't on the other hand, be useful to determine the views of developers as a whole.

Quick conclusion

Some of these small scale studies remind me of something attributed to 19th Century British Prime Minister Benjamin Disraeli -- "There are three kinds of lies: lies, damned lies, and statistics."

While the results of small scale studies are interesting and may be representative of a specific audience, it would be wise to have an understanding of who sponsored the study, how the sample was gathered, how their views were determined, and how their answers were analyzed before taking the results to heart. A study sponsored by a supplier that questions only friends and family of that supplier may not offer a broad view of market requirements. If your organization fits within the boundaries of the survey sample, the questions asked seem to represent the type of questions your organization is considering and the analysis of the answers appears to be neutral and free of bias, the results might be quite useful.  If not, I'd urge that you put the results in the "interesting but not very useful" category.

In the case of the study Blue Coat Systems presented, it is hard to know many things about the study itself just from the press release.  The folks at Blue Coat Systems are good people and so I would guess that they're probably just trying to help the market understand that the products and services they're offering meet the requirements of a growing list of organizations.

When other studies are considered, it would be wise to be skeptical.