Are surveys executed at conferences useful or representative?

Surveys and focus group studies help suppliers better understand what customers want, their priorities, their sensitive to cost, their view of the future and can be of great value. Focus group studies are useful when the analyst doesn’t really know the right questions to ask. Surveys are useful when the questions are known.

These tools, like any other sharp instrument, have to be used very carefully, that is held by the handle not by the blade, or they can hurt more than they help. A key to the gathering of useful information is to make sure that the responses are representative of the audience being studied. This means careful creation of questions so that they are not biased, do not lead the respondent to specific answers and ask the right questions to tease out useful information. It also means obtaining responses from a representative sample. So:

• Surveys done at conferences are often questionable becuase they may only represent people who chose to come to the conference rather than a broad audience.
• Surveys composed of respondents only from one region may not be broadly applicable when worldwide opinions are needed
• Surveys that clearly show a bias towards or against a company are not likely to produce useful results

What’s the point, Dan?

Recently I received the results of a survey that I was told was executed by EnterpriseDB at the 2010 JavaOne conference. I was advised that the survey results were based on responses from more than 600 JavaOne attendees. I was told that the survey reveals that people think Oracle is bad for Java and MySQL -  there is specific trepidation about open source project stagnation under Oracle’s watch, not to mention expectation of services / support price hikes.

I, of course, wondered what the real motivation was for the survey and if it was a real attempt to learn something about how people thought and felt about the future of Java or simply a cheap attempt to put forward the views of EntepriseDB. So, I asked about the make up of the survey respondents and to be shown the survey instrument.

I wanted to know if the respondent population was representative of the market as a whole or only a portion of those who attended the event. I also wanted to know if the questions were seriously trying to learn what people were thinking or was an attempt at promoting a certain point of view.

Here is a segment of the survey instrument I was sent:

1. Under Oracle, open source projects will:
   • Thrive
   • Barely hang on
   • Die
2. Do you think the Oracle lawsuit against Oracle is:
   • Good for Java
   • Bad for Java
   • Has no impact on Java
3. At renewal time, do you expect MySQL prices to:
   • Increase
   • Decrease
   • Remain the same
   • I use PostgreSQL (not MySQL)
4. Who would you like to see in a dunk tank:
   • Larry Ellison
   • Bill Gates
   • Steve Jobs
   • Eric Schmidt

Here are highlights of the survey results that were reported

• Open source projects will barely hang on: A vast majority of respondents (46 percent) say they expect open source projects such as MySQL to stagnate under Oracle ownership.
• Oracle is bad for Java: 56% of respondents indicated increased concern with Oracle’s lawsuit against Google regarding copyright and patent infringement charge relating to Java believing the lawsuit is bad for Java. Only 13% said that the lawsuit would be a good thing for Java.
• MySQL process to increase at renewal time: Under Oracle, more than 42 percent of respondents say they expect MySQL prices to increase at renewal time.
• Respondents prefer to see Larry Ellison in a dunk tank: Nearly 40 percent of respondents would like to see Larry Ellison, CEO of Oracle, in a dunk tank more so than other well known executives like Bill Gates, Steve Jobs or Eric Schmidt.

Here’s what the promoters of the survey had to say about the results

At least from this unscientific survey, the results seem to indicate the IT community is weary about Oracle’s plans with open source projects like MySQL and Java. More and more people today are demanding choice and flexibility without vendor lock-in, so the intensity of the skepticism around Oracle’s stewardship of open source from this survey is actually not surprising to the folks at EnterpriseDB.

Snapshot analysis

It is clear that this survey was designed for its entertainment value not to get a clear idea of what conference attendees were thinking. It also is quite clear that the goal was putting forward EnterpriseDB’s views rather than finding out more about the views of the conference attendees. This causes me to discount the findings and say that they are really not all that useful.

It is clear to me that this really was an attempt by EnterpriseDB to put forward their own agenda, one supporting PostgreSQL and against Oracle and MySQL, rather than an attempt to find out what JavaOne attendees were thinking about the future of either Java or MySQL.

I also feel that surveys such as this one are part of the reason that it is increasingly difficult to get good information about the thoughts and requirements of decision makers. Potential respondents have come to see all surveys as frivolous, disguised marketing tools, or designed to make a point rather than to get useful data. So, many refuse to take part.
It’s sad that a company like EnterpriseDB went to this effort to make their point. EnterpriseDB, you hurt your cause more than you helped it this time.