Getting caught under a survey avalanche
Press releases touting one survey or another cross my desk nearly every day. This time, the study came from a supplier of cloud-based email backup and archiving services. Although the responses offered a few useful insights, I was left with the impression that the whole point of the survey was to drive potential customers to the company's website to purchase the company's services.
True research or marketing tool?
From time to time, the surveys mentioned in the press releases are well thought out, well designed and well implemented attempts to learn what IT decision-makers are thinking. Unfortunately, it is far more common for them to be self-serving, badly designed, vendor-sponsored attempts to demonstrate that the company's product strategy or a specific product are wonderful and their customers love the company.
Regardless of how small the sample is or how badly the survey instrument was designed, the results are trumpeted as being representative of the industry's thinking.
Studying the obvious
Some of the time, these surveys are really painful studies of what appears to be obvious. Although the research was well designed and field work was well executed, they don't pass the reasonable person test. That is, what would a reasonable person, who was not being incented based upon the results of the study, think about this topic.
Surveys are a sharp instrument
Like any other sharp instrument, surveys need to be used very carefully — that is, held by the handle not by the blade, or they can hurt more than they help. A key to the gathering of useful information is to make sure that the responses are representative of the audience being studied. This means careful creation of questions so that they are not biased, do not lead the respondent to specific answers, and ask the right questions to tease out useful information. It also means obtaining responses from a representative sample.
Unfortunate problems
The problems I have with quite a number of the marketing-focused surveys that come across my desk usually fall into two categories:
Is the sample representative of the industry at large or, at least, the market segment it is supposed to address? That is, are the right people being asked?
Quite often, a small sample is used and broad, global statements are made. Examples would be something such as the sample is made up of respondents from a single country and don't represent the worldwide market; the sample is made up of representatives of a single market segment and don't, as before, represent the worldwide market; or the sample is made up of only the sponsor's customers and/or the survey was conducted at the sponsor's own customer event, making the sample quite limited.
The sample purports to present what companies are planning to do, but the sample may not include company decision-makers.Featured
The survey instrument is biased or leading, making the results questionable. That is, are the right questions being asked and are they being asked in the right way?
The questions might assume a given position and make no provision for respondents to make contrary answers. This is the "when did you stop beating your spouse?" type of question. There is no provision for someone who does not beat his/her spouse to respond.
The questions might be biased to support the need for the sponsor's product or service. So, of course, the questions would assume the need for this type of product.
The survey instrument asks specific questions for which respondents don't have answers. If the survey offers "don't know" or "not applicable" as choices, that is far better than offering a choice that asks respondents to estimate (or make up) answers.
When analyzed, these studies can be seen for what they are: marketing tools designed to promote the company, its views and its product or service offering rather than trying to find out what IT decision-makers are thinking.
Although there is no way to know this, it appears that some suppliers turn to funding survey-based research when they don't have exciting technology news or a new product to launch. It appears that they hope that the press release will be widely mentioned in the media and their current portfolio of products and services will see a boost in popularity.