AIX beats Solaris!

Here's a typical bit:

HP and IBM basically tie on three of the above criteria - Operating System Quality, System Management Suite and Partitioning/Virtualization Features. IBM notches a clear victory in Operating System Features. While IBM and HP battle it out, Sun isn't much of a factor.

And here's the real summary:

Sun had the worst showing in the survey. They were third in almost every category, often a distant third. These results are important as they signal that Sun's immense installed base may be vulnerable to predations from their better-funded competitors. (Emphasis added)

So how did results like these come about? Well, the study gives a bit of information about its respondents, goes into detail on the weighting scheme used, and generally gives the impression of having been fairly done. Here's a sample:

We asked 197 enterprise UNIX customers about their experiences and their views of the three major UNIX vendors. Our survey was aimed at actual data center personnel, data center managers, system architects, etc., rather than those at the CIO and CFO level. We have found that people on the data center floor have a much better idea of what works well (and not so well) in their infrastructure and are generally not shy about expressing their views - both positive and negative

...

80% of surveyed data centers have servers from two or more vendors, with almost half of the survey population running servers from all three major manufacturers.

Unfortunately the report does not describe the survey's response in terms of either real numbers or percentages. Instead almost everything is couched in terms of a metric the authors call a Vendor Preference Index or VPI. Here's the rationale:

Since there are only three major UNIX vendors (Sun, HP, IBM), and Sun has by far the largest installed base, we normalize the data so that no vendor is advantaged (or disadvantaged) by the sheer size of their installed base. (Empahsis added)

A VPI is calculated as the number of "votes" a vendor gets on a question divided by the number of respondents who had claimed a primary corporate commitment to that vendor's Unix.

Imagine, therefore, that we have this situation:

According to the report these results mean that IBM clobbered the competition with Sun placing a distant last.

In reality, however, 61% of the respondents prefered Sun, 22% prefered HP, and only 17% percent actually voted for IBM - meaning that Sun clobbered the competition with IBM placing a distant third.

What's going on here is that the VPI calculation produces a ratio that looks like a percentage but isn't one because the two quantities: the number expressing a preference for the vendor on some question, and the number of respondents who cite that vender as their primary supplier, are only indirectly related.

The 0.85 VPI score calculated above, for example, does not mean that 15% of Sun's users voted for someone else; in fact, it's not possible to ascribe a precise meaning to it at all.

What should be obvious, however, is that the VPI measure can be used to cook the books on a user survey simply by selecting a plurality of respondents from among those describing themselves as predominantly, but not solely, in the camp you want to damn and consequently as few as possible from the camp you want to praise.

What happens when you do that is driven from the fact that people generally want to to see themselves as fair: meaning that someone running Sun and IBM servers in the same data center isn't usually going to answer "Sun" to twenty or more questions in a row. So if 70% of the respondents classify themselves as primarily Sun users and 15% of them mutter "IBM" in response to at least one question, the VPI calculation will make Sun look like a loser and IBM look like a winner.

Oddly, the less real information there is in those "IBM" responses, the more nearly random the distribution of their "IBM" answers to the questions will be: meaning that the less representative the responses actually are, the broader IBM's victory will appear to be.

In other words, this is actually a very cool piece of work - but readers should understand that the larger the margin this report awards IBM on any one question, the greater the margin by which Sun actually won it.