Gartner Magic Quadrant lawsuit: Sour grapes or real gripes?

Gartner Magic Quadrant lawsuit: Sour grapes or real gripes?

Summary: Industry analyst firm, Gartner, is the target of a lawsuit from a software vendor challenging the "legitimacy" of Gartner’s Magic Quadrant rating system. The suit has brought forth an array of divergent opinions.

SHARE:
TOPICS: Legal, CXO, IT Priorities
12

Industry analyst firm, Gartner, is the target of a lawsuit from software vendor, ZL Technologies, challenging the "legitimacy" of Gartner’s Magic Quadrant rating system. The suit has brought forth an array of divergent opinions.

Background. As one of the top analyst firms, with revenue in excess of a billion dollars, Gartner's opinions and recommendations carry substantial weight with technology buyers and influencers.

On a special website page devoted to the lawsuit, ZL Technologies claims that Gartner's Magic Quadrant does not present a fair and accurate portrayal of the software market. The company says:

Gartner’s use of their proprietary “Magic Quadrant” is misleading and favors large vendors with large sales and marketing budgets over smaller innovators such as ZL that have developed higher performing products.

Here is the original legal filing:

ZL v Gartner Complaint

Gartner's legal response is highly technical, but states that Magic Quadrant reports are "clearly opinion" and therefore protected by the first amendment right to free speech. See page ten of Gartner's Motion to Dismiss the ZL complaint: Gartner Motion to Dismiss

The four legal documents made available to date are available by clicking here.

Divergent opinions. Lawsuits often involve strong emotions and polemic, and observers in this case have divided opinions.

In the anti-Gartner camp, ZDNet blogger, Dennis Howlett, quotes a third party questioning Gartner's objectivity and integrity:

‘And you know what, it doesn’t stop at tin cupping. They double dip. They say they’ll help you with strategy, charge you for the pleasure and then go on to sell analysis based on that to say exactly the opposite.’

Howlett adds:

[T]here is no denying that Gartner opinion stated as fact has become a powerful tool for vendors to wave in front of customers.

Taking a different view, Mark Logic's CEO, Dave Kellogg, blogs:

“Sour grapes” spring to mind as an immediate reaction. In fact, ZL concedes that they’ve been ranked in the “niche” segment of every email archiving quadrant since 2005. (Ouch!) But they nevertheless argue that bigger stakes are in play and that this is not only about ZL, but Gartner itself, technological innovation, and very nearly preservation of the American way of life.

THE PROJECT FAILURES ANALYSIS

The lawsuit raises important issues about the role of industry analysts in the enterprise software ecosystem.

Analysts help technology buyers and sellers understand the market environment in which both groups operate. This intelligence helps technology vendors determine the strategies and features that are important to buyers. Likewise, analysts guide enterprise buyers through the challenging process of procuring large, expensive systems. When this system works properly, analysts provide a valuable service.

Carter Lusher, strategist at analyst relations consulting firm SageCircle, believes reputable analyst firms strive for neutrality, but don't always communicate limitations of their research. He told me:

Savvy research consumers use a variety of sources for information and advice. Unfortunately, consumers of analyst research don’t always know how to interpret reports such as Gartner’s Magic Quadrant. Analyst firms should do a better job educating readers.

Other observers, such as longtime industry analyst Jonathan Yarmis, question the relationship between commercial contracts and favorable analyst coverage. When Gartner analyst, Thomas Bittman, wrote a blog post staunchly defending the analyst business model, Jonathan responded with a strongly worded comment:

If you think all that money vendors are spending is purely because of your insights, again you're being naive. There's an influence component and an insight component. Why do you think they spend so much more with Gartner? Are your insights really that much keener? Or are the vendors more concerned with your market reach and therefore the need to more strongly influence you compared with everyone else?

I spoke with Jonathan and asked him to elaborate:

Most analysts have integrity and I do not believe that a commercial relationship with vendors overtly influences analyst opinions. However, it is also true that money buys access, which, deftly used, can lead to improved favorability.

My take. Analyst research and reporting is not an exact science, which does lead to real or perceived conflicts of interest. The analyst industry can reduce potential conflicts by improving transparency around how it forms opinions and makes recommendations.

Analyst clients should ask firms for spreadsheets and other supporting documentation that explain how vendor and product rankings are determined. Although differences of opinion will always exist, transparency helps ensure impartiality.

To increase transparency, analyst firms should also disclose their revenue relationships with vendors. How much do they get from whom, and what is the break down between retainer and consulting revenue? Consulting is much more influential than subscription research or retainer arrangements, because it represents one-on-one relationships with an individual analyst.

Regardless of who wins or loses the lawsuit, it is beneficial to shine the light of day into this important segment of the technology industry.

Topics: Legal, CXO, IT Priorities

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

12 comments
Log in or register to join the discussion
  • Merits of Lawsuit

    While Gartner may have its own objectives, I fail to see how the lawsuit has any merit. Should Michigan State sue US News and World Reports because the school isn't listed as a top 50 university?

    These are opinions. Yes, they are influential but they remain opinions.
    phil_simon
  • RE: Gartner Magic Quadrant lawsuit: Sour grapes or real gripes?

    There has been a longstanding view on why Gartner does not provide
    details for their rating criterion as Forrester Wave Does.

    I feel the Excel tool provided by Forrester with all the
    evaluation criterion and why a vendor was scored in that manner
    very objectively is much more transparent. Further the tool lets
    you change the weighting to suit your own needs.
    bullonthemove
  • RE: Gartner Magic Quadrant lawsuit: Sour grapes or real gripes?

    It's not just the "official" analysts who give top of mind to those vendors with whom they have the deepest relationships. Everyone who consults/reports/blogs/kibbitzes about a particular business domain and the related technology providers has limited mind-share and devotes more of that mind-share to those vendors/providers about which they know more and/or with whom they interact more. With only so many hours in the day, and more vendors in any one space than can be followed with equal attention to all, we all do triage with our limited mind-share, and one element of that triage equation -- consciously or unconsciously -- is the extent of our relationship with/interest in a particular vendor/provider. One of the reasons that my consulting contract stipulates that will I tell end-users about my relationships with vendors/providers is to ensure that any potential "excess of mind-share," never mind actual conflicts of interest, are completely transparent. With that information, it's up to my clients, all of whom are adults, to decide to what extent my greater knowledge of some vendors/providers -- the good, bad and the ugly about them -- is relevant to the matter at hand.
    naomibloom@...
    • Transparency

      Naomi,

      Thanks for sharing this excellent point. Any so-called "influencer" is subject to the same dynamics described in the post. As you described, openness and transparency prevent many problems and help maintain trust.

      mkrigsman@...
  • RE: Gartner Magic Quadrant lawsuit: Sour grapes or real gripes?

    Any vendor not in the "magic" quadrant knows how hard it is to move "up and to the right" because they don't have the marketing dollars to spend to get analyst mindshare.

    Yes, what they say is opnion-based but when prospects only looks at the vendors in the top right it's a huge competitive disadvantage - and it's getting worse.

    Gartner needs to separate the consulting business from the vendor evaluation business and they need to add a lab component to do some independent testing when doing evals instead of just grading vendors according to their "opinion".
    techytalk
  • Sour Grapes and Real Gripes

    Beyond the plain rights of the lawsuit, it seems that this post is addressing four issues:

    * Opinion vs. hard fact

    * Transparency vs. hidden bias

    * Being (or not being) in the "in" crowd

    * The power of influence

    Opinion is someone?s appraisal or judgment of both facts and subjective things. Facts are data such as something is 2.5 inches high; 2 is less than 2.25. At some point, the facts get interpreted according to set of values when you want the data or the synthesis of the data to mean something. It?s subjective. This is why different people can arrive at different conclusions based on the same data. Once an opinion gets rendered, somebody some where, will disagree with the finding.

    Plenty of people have commented on the need for transparency so that others can understand what data was collected, and whether unfair bias was applied in the rendering of an opinion. So I will say no more.

    Most people aspire to be in the "in" crowd. Those on the "out" want to challenge the exclusion either by wanting to disavow the specialness of the in crowd, or the entry criteria. But once they break in to it, they defend it. Chances are that ZL Technology wouldn?t be complaining if they were deemed a Magic Quadrant company.

    Lastly, the power of influence that big name analyst companies and analysts have and all the attendant issues.
    elizab
  • Yet we all strive to be in the Quadrant, right :)

    Considering Gartner don't hold back on also raising
    negative points on the companies in the TOP RIGHT of the
    quadrant, I don't see what the problem is. It is clearly
    subject matter as part of a review process. Our company
    hasn't made a single magic quadrant from Gartner, other
    than cool vendor and to be honest, OpenSpan has sold to
    some of the largest institutions in the world! Go figure.

    CIO's don't make their ultimate decisions on what is in the
    quadrant. It's just one part of the review process.

    It is for the company to grow on it's own merits and truths. Support
    from Analysts like Gartner and Press and reviews is very important but
    not the only piece.

    My 2 Cents,

    Francis Carden, Founder, OpenSpan
    francis.carden@...
  • RE: Gartner Magic Quadrant lawsuit: Sour grapes or real gripes?

    As an Enterprise software consultant I have been skeptical of Analyst reports in general, and particularly the Magic Q, for years. The conflict of interest are rampant and obvious to all except the CIO and CTO who use it as a crutch or a hedge when making often multi-million dollar purchasing decisions. The perpetuation of this corrupt system is solely the responsibility of the consumer of the service. It is frustrating to everyone in the know, and should be exposed and cleaned up.
    johnnykessel
  • Interesting response from Gartner...

    By trotting out the "hey, it's just an opinion" argument, Gartner may win this case at the expense of losing standing in the marketplace. Gartner, by not even attempting to argue that their opinion has merit, seems to have conceded that their opinion in this particular case is suspect at best. Had they been making this case, the question of "sour grapes or real gripe" would seem to be a good one. As it stands right now, Gartner has given the argument for "real gripe" a strong leg up. I have always said that the arguments you choose to use will ultimately tell more about your case than anything. The argument "hey, it's just opinion" weighed against the argument "hey, it's opinion based on fact and I stand by it" tells me that Gartner knows that the facts don't bear out the opinion they are selling.
    jasonp@...
  • RE: Gartner Magic Quadrant lawsuit: Sour grapes or real gripes?

    What's silly here is that it's even a topic of debate (the neutrality / legitimacy of the analysis). Gartner spends more time with companies that spend more money with them. Period. Does the money that changes hands allow those companies to "magically" rise in the report? Duh...

    A few other questions about the legitimacy of these paid advertisements. #1- how can you rate technical products in the absence of actual lab testing? #2- how do analysts who have never worked in a datacenter and have no "hands on" experience accurately place products they have never used in proper context? #3- how many end users actually READ the analyst reports (there are never third party verified studies about the actual readership of the reports).

    Sour grapes or not, this type of lawsuit is good for the industry if it gets people asking questions.
    travisvan
  • RE: Gartner Magic Quadrant lawsuit: Sour grapes or real gripes?

    Really? So all the events they host are free? And mega vendors that shell out big bucks to them don't get more face time (and time on the phone)? Pretty naive ...
    travisvan
  • some additional points

    I'm a former Gartner analyst (as is Carter
    Lusher, which I didn't see mentioned in the
    article).

    You know, there's nothing 'ouch' about being a
    niche vendor. Just like not every movie is a
    blockbuster, but some do very well for a
    particular type of audience, 'niche' simply
    means a product that's more focused on a
    particular area.

    Another major component of Magic Quadrants are
    user interviews, not just associated with the
    MQ itself but which are performed constantly,
    year-round.

    But yes, this lawsuit sounds like sour grapes
    to me.
    slfisher@...