Gartner in the dock over Magic Quadrant

Gartner in the dock over Magic Quadrant

Summary: Friday October 23rd will see Gartner argue a motion to dismiss a complaint by ZL Technologies Inc about the famed Gartner Magic Quadrant. According to court papers, Gartner will argue to dismiss based on First Amendment rights citing that the Magic Quadrant is not meant to represent statements of fact but is based on pure opinion.

SHARE:

Friday October 23rd will see Gartner argue a motion to dismiss a complaint by ZL Technologies Inc about the famed Gartner Magic Quadrant. According to court papers, Gartner will argue to dismiss based on First Amendment rights citing that the Magic Quadrant is not meant to represent statements of fact but is based on pure opinion.

I'm not qualified to argue the legal ins and out of the case and so if anything I say here veers in that direction then you can safely ignore it. However, ZL's plaint is not really about constitutional rights but about what it sees as an abuse of market power.

In a statement sent to me, ZL says of its original plaint:

ZL claims that Gartner’s use of their proprietary “Magic Quadrant” is misleading and favors large vendors with large sales and marketing budgets over smaller innovators such as ZL that have developed higher performing products. The complaint alleges: defamation; trade libel; false advertising; unfair competition; and negligent interference with prospective economic advantage.

This is a topic that from time to time gets aired by those who believe Gartner (and other industry analysts) are in what some regard as an incestuous 'pay-to-play' arrangement with vendors. Earlier in the month, Gartner analyst Tom Bittman hit out at those who argue 'pay to play:'

As an analyst at Gartner, I can’t describe how angry I get when I read bloggers spouting as “fact” their opinion that I and my teammates have no integrity. That we can be “bought.”

In my 14+ years at Gartner, I have never, ever allowed a vendor to influence my opinion with anything but facts. Period. They have certainly tried to influence me with non-facts. I can say this definitively – it has never worked.

I don’t think there is a single vendor that I have dealt with who has not been very angry with me at some point. Tough. I’ve been yelled at by many IT executives – including the CEOs of Microsoft and HP, and many other firms. I can’t think of one of those cases when I changed my analysis one bit. I can’t speak for other firms, but at Gartner, getting yelled at by a CEO is a badge of honor. Being proven right as time goes on – priceless.

I certainly spend time helping vendors with their strategies and their marketing messages – and I enjoy doing it. Frankly, the ones who yell at us the most seem to respect our opinion the most. We can spot holes a mile away, and engaged early enough, we can help vendors fill those holes with real product offerings – that not only help the vendors, but help our end user clients. And my primary business is helping end users.

The post attracted 33 comments including several from Vinnie Mirchandani and myself. Vinnie argues pretty much the same thing that ZL is saying (you need to check the post as there are no comment links):

Tom, as for your former colleague and now a blogger I guess I see both sides. If you think it is just bloggers raising the issues, you are not reading the market, including your customers well.

The issue is not at an individual analyst level. It is at the firm level. You know it and I know it that vendor sourced revenues have been growing nicely as percentage of Gartner revenues. You and I know the majority of top 25 Gartner revenues are vendors like IBM – and individually they dwarf what even the biggest user entities like GE or BP pay you.

It would help if Gartner was transparent about that, rather than avoid talking about it, and if Gartner aggressively showcased how those vendor interests and subtle and not so subtle pressure from them is balanced with user inteersts. The answer usually is the Gartner Ombudsman blog does that. Seriously?

The other thing is 2 of Gartner’s best known tools – the MQ and the Hype Cycle are skewed towards larger vendors and against startups. The MQ rewards market share and viability or tenure in a market. By its very definition the Hype cycle warns people about newer stuff. So Gartner comes across as pro-establishment with more skew towards larger, incumbent vendors

In the field I see time and time again vendors using your metrics where it suits them. I see economics that are dated. In a consulting assignment last year, the CIO asked me fairly loudly during a presentation when a vendor put some of your metrics “so is that 30 or 40% over priced?”

[my emphasis added]

Tom responds by arguing that the MQ is more scientific these days:

In your day, MQs were developed by smart analysts placing dots. Today, we have a much more rigorous mathematical model – we don’t just place dots. It isn’t perfect, but it’s better. “Ability to execute” includes viability, which will tend to favor large, incumbent vendors. But “vision” actually favors the start-up. When we execute the MQ well, you clearly see the visionaries, the incumbents, etc. It is in our best interest to show that variety in an MQ.

Just because something is more scientific doesn't make it fact as any disinterested observer of the climate change debate will tell you. But there is no denying that Gartner opinion stated as fact has become a powerful tool for vendors to wave in front of customers. I argued that:

@tom: I get complaints pretty much every week of the ‘pay to play’ argument so whether you believe it or not is immaterial. It goes back to what @vinnie says about firm level issues and the corporate emphasis on aggressive selling – or as one of your major clients puts it to me: tin cupping.

I appreciate and respect that individual analysts try maintain their independence and I see your examples. I know several of your specialist analysts in my specialist space and I would not doubt their personal integrity.

It's worse. In one conversation last week, a senior firm's representative went on to add: 'And you know what, it doesn't stop at tin cupping. They double dip. They say they'll help you with strategy, charge you for the pleasure and then go on to sell analysis based on that to say exactly the opposite.'

In another comment, Jonathan Yarmis was far more blunt:

I’ve been on all sides of this argument (Gartner analyst, non-Gartner analyst, AR professional) and to argue that vendor money does not influence perceptions and outcomes is hugely naive. My standard spiel as an AR practitioner was that money did not necessarily equate to influence. I’ve seen people spend a lot of money to no apparent benefit. I’ve seen people spend no money and get great outcomes. But money well spent equates to more positive outcomes.

If you think all that money vendors are spending is purely because of your insights, again you’re being naive. There’s an influence component and an insight component. Why do you think they spend so much more with Gartner? Are your insights really that much keener? Or are the vendors more concerned with your market reach and therefore the need to more strongly influence you compared with everyone else?

So, the vendors are spending because they believe they’re influencing you. You’re claiming you can’t be influenced by this. Either you’re wrong, and there is an influence element at play, or they’re wrong and they shouldn’t be spending nearly as much money with you. Which is it?

It is for these and other vendor related issues that Vinnie, Ray Wang, Frank Scavo, Oliver Marks and I decided to establish Enterprise Advocates. For too long the voice of the customer has been sidelined in favor of market driven analysis that in our view is massively skewed towards the 'establishment.' The vendor pieces may move around on the chessboard of enterprise computing but the established names are pretty much always there. Go to any trade show and you'll see hordes of analysts being shuffled off into side rooms for 'exclusive' briefings, sometimes under NDA, sometimes not.

To the credit of a handful of vendors, they have chosen to embrace the independent blogger/analyst voice but there is always pressure upon us to tone down. It goes with the turf. SAP has the best developed blogger program of any software vendor. That's one reason why you hear so much about them in this blog. That doesn't prevent me getting the late night calls asking why I said this or that. It sure as heck doesn't prevent at least some executives trying to tear us a new one when we bang on about maintenance.

Others try and ignore us but what they don't realize is we're not going away. Our clients demand that independence. It's what they pay for. Simply trying to shut us down is counterproductive. That doesn't stop the established hand wavers from boosting technologies but that's OK too. Practical experience and direct customer experience wins over marketing when it comes to signing the check. That's perhaps where ZL would be best diverting its legal spend.

Even so, ZL points up some uncomfortable assertions. Even if it lucks out in court it will have been successful in drawing attention to important issues. As the ZL statement said, it hopes to highlight:

- Fair Disclosure on Conflicts of Interest – Gartner generates its revenues from payments made by the same vendors whose products it evaluates. Similar to the new rules now being imposed on financial ratings agencies on Wall Street, Gartner should be required to disclose the revenues received from the vendors it ranks. - Fair Disclosure on Evaluation Scores The tech industry would benefit if Gartner were required to disclose more data in its evaluation process and disclose component scores so vendors know exactly where they are lacking and by how much and take corrective action. Currently, there is zero disclosure, which can lead to arbitrary placement, with no recourse and no basis for appeal. - Better OversightGartner currently has an employee act as ombudsman to handle disagreements. The conflict of interest is self-evident in the way ZL’s concerns were summarily dismissed with little supporting evidence. There is a crying need to establish an impartial ombudsman similar to those found in public media, in order to ensure purchasers that they are receiving impartial analysis.

These are all points I welcome.

I spend a lot of time talking to other so-called analysts. Almost to a man/woman (but not entirely) they are 70-80% in the pay of the vendor community. Many believe they are independent. However when I ask who really pays the bills they go silent. Shout all you like about defending your independence but at the end of the day? Show me the money.

Would I work with a vendor? Of course. It's inevitable that vendors need the buy side and where that requires a consulting component then there needs to be a transaction. But it is always on the strict understanding that what I am saying is from the buyers' perspective and that I am not here to defend anyone's marketing. It also means I will strictly limit the amount of paid for assignments coming from that side of the 'house.' Oh yes - and any of those assignments will be disclosed. Promise.

Topics: Banking, Browser, CXO, Enterprise Software, IT Priorities

Dennis Howlett

About Dennis Howlett

Dennis Howlett is a 40 year veteran in enterprise IT, working with companies large and small across many industries. He endeavors to inform buyers in a no-nonsense manner and spares no vendor that comes under his microscope.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

32 comments
Log in or register to join the discussion
  • Persona Non Gartner

    Over the years, I've watched, participated in researching and experienced the results of the positing of Gartner's disclosures. My analysis is as follows:

    On any given subject at any given time, there is a .90 probability that the opinions expressed by Gartner will be either flat out wrong or miss the mark badly.

    We hired Gartner to make recommendataions and paid a hefty sum. Good thing we ignored them.

    The best use for the Gartner Group is for techs to manage to get them, make a presentation to management -- to make the points and use persuasion to advocate what we all know is the right thing to do [but management, being too dense to understand the issues, will listen to "experts" rather than to those they have hired to do the job].

    Otherwise, I'd advise seeing what Forrester has to say because it's usually more sensible.

    The most frightening thing is that Gartner has the ear of United States Congressmen. Ackkk!
    dbecker@...
    • Ouch!

      ...but let's not forget that managements like 'ticks in boxes.' Makes them feel safe at having a deflecting finger to point.
      dahowlett
    • "Gartner . . . flat out wrong . . ."

      "... or miss the mark badly."

      Like when they averred that "subscription-based music" site sales would soon overtake those on sites like Apple's iTunes Store?

      That was over five years ago, back when Real's CEO Glaser and somebody at Gartner were apparently good buddies (or something).

      Ha ha ha ha.

      Can you say MISS THE MARK?
      brian ansorge
  • RE: Gartner in the dock over Magic Quadrant

    To Tom Bittman?s Integrity. You said ?Today, we have a much more rigorous mathematical model ? we don?t just place dots? on MQs. Unfortunately, your employer has told the court the MQ report is "pure opinon," i.e., NOT BASED ON FACT AT ALL. To hide behind the First Amendment, your employer appears willing to throw the analysts? work under the bus. You said you?d be ?outta there? if you felt Gartner?s integrity ?bending? ? what now?
    konleong
    • You're confusing fact and opinion

      konleong,

      Gravity is a fact. The how and why of gravity is largely opinion. Evolution is considered to be a fact. The how and why of evolution is opinion.

      Apparently you believe 'pure opinion' cannot be based on facts. I'm not going to explain to you why you are incorrect. That is your homework assignment for this week.
      josephmartins
    • ZL media stunt perhaps? - Magic Eight Ball says "Signs point to yes"

      Please tell me you're not the CEO of ZL by the same name?

      I'm not fond of Gartner Mr. Leong, but let's not distort reality with your poor reasoning skills. The following is the complete context of Gartner's argument.

      From Section II of Gartner's response to ZL's Opposition to the Motion to Dismiss:

      "It is undisputed that an expression of pure opinion is protected by the First Amendment
      and may not form the basis for a civil law suit. See Partington, 56 F.3d at 1153. A pure opinion is one that does not imply facts capable of being proved true or false. Id. at 1153 n.10. To determine whether a statement constitutes non-actionable opinion rather than an implied assertion of objective fact, the Ninth Circuit applies a three-part analysis considering (1) whether the general tenor of the entire work negates the impression that the defendant was asserting an objective fact, (2) whether the specific content and context of the statements, including the use of figurative or hyperbolic language, negate the impression of an assertion of fact, and (3) whether the statement in question is susceptible of being proved true or false. Id. at 1153-60; see also Knievel v. ESPN, 393 F.3d 1068, 1072 (9th Cir. 2005) (same). Applying each somewhat overlapping element of this analysis to the Magic Quadrant reports demonstrates that Gartner?s alleged statements are non-actionable opinion."

      Gartner's attorneys continue in subsections A, B and C with a very well-reasoned response.

      All other BS and phallus-waving (from both sides) aside...ZL hasn't a legal leg to stand on. People may not like to hear that, and it's not going to earn me any brownie points, but that's reality.
      josephmartins
  • My two cents

    Dennis,

    I've already tackled many of the points discussed in your post elsewhere on the Internet. So I will focus on ZL's beliefs and intent in this comment.

    [Note: Before I begin let me say that the terms 'vendor' and 'end-user' create artificial boundaries where none exist. A vendor is nothing more than the sell-side of a company. And a buyer or end-user is nothing more than the buy side of a company. There is nothing more amusing than listening to a Ford, GE, MIT or Warner Brothers IT employee talk about 'those damn lying vendors' as if he does not draw his paycheck from one. Let us not forget how our bread is buttered.]

    Now...

    Companies large and small (from the IBMs to the ZLs) are a dime a dozen regardless their size. Each of them believing their products to be more innovative, their employees smarter, and their road maps more visionary than those of their competition. They are, after all, drinking their own kool-aid. Everyone else is, in their minds, delusional.

    In these days of incredibly complex hardware and software products, much of the analysis - while it may be based on some facts such as speeds and feeds - is largely subjective opinion AND largely contextual.

    ZL is pissed off because some group of analysts at the mighty Gartner didn't rank it well in their largely subjective MQ? So what. They are entitled to their opinions. If the product is as innovative as ZL believes it to be, surely other analysts at other firms would/will have a different opinion.

    Yes Gartner's opinion differs from ZL's. And, yes Gartner does earn money from the buy- AND sell-side of its clients. What of it? Are we really foolish enough to engage in circumstantial ad hominem reasoning? That's naive and indefensible.

    ZL is implying that the MQ is biased solely based on Gartner's revenue sources. The company clearly hopes that disclosure would create enough doubt about MQ among Gartner followers to decrease the MQ's value and perhaps elevate what the company believes to be its own superiority.

    [Note: I should mention I could care less about the MQ. I prefer to dig up my own evidence and draw my own conclusions.]

    Disclosure aside (and I believe disclosure is a good thing), no matter how we slice it ZL's complaint is nothing more than kool-aid driven smacktalk.

    Regarding your new adventure, Enterprise Advocates, I'll reserve my opinion about it until a later date when more information is available. Those who know me well know how I feel about self-branded 'end user' advocates. I often refer to them as wolves in sheep's clothing.
    josephmartins
    • Cynics abound

      @jospehmartins - I don't know you so it would be helpful to have some links to what you are saying so I can understand better what you are saying.

      I must say you are describing a world I don't recognize. As someone who has batted for the buyer the last 30+ years, the notion that buyers and sellers are part of the same equation is logical nonsense. To equate IBM and ZL in the same breath is entertaining but ultimately nonsensical. Although I note the IBM and Canonical tie up as an interesting move. Big Shark meets Feeder Fish?

      By all means judge Enterprise Advocates by our actions. If we fail to deliver on our promise then we deserve all the flames that get thrown at us. In the meantime - watch this space and feel free to hear what we say the day after tomorrow about SAP maintenance. Need a link then ping me directly - would love to hear your questions.
      dahowlett
      • Yes, yes they do.

        Visit StorageMonkeys, Dennis. Those are among my most recent comments. And you'll get an eyeful there.

        Like it or not, buyers and sellers ARE part of the same equation. The fact that you refer to it as 'illogical nonsense' is worrisome to say the least. If you're not spelling that out to your clients then you're doing them a disservice. I find that many 'user advocates' advocate that same ill-informed perspective.

        Regarding IBM and ZL, suffice to say it is not nonsensical to suggest they both drink heavily of their own kool-aid. I've been listening to, and reading it for years.

        I'm not interested in what EA does. What will interest me most is what it doesn't do. Are you prepared to tell your clients what they are very likely not going to want to hear about themselves and their operations? Time will tell.
        josephmartins
        • What's your problem?

          I've tried to visit StorageMonkeys but it tells me nothing I don't already know. I/(we) act for buyers - end of story. If you have a problem with that then kindly explain why rather than trying to change the topic by skewing implied dishonesty to your own agenda.

          If you don't care about EA then fine but kindly restrict your argument to the veracity of the story.

          I can tell you as a matter of 30+ years FACT of dealing with buyers/sellers, they are absolutely not part of the same equation as you describe and in any event have nothing to do with the analyst story I have written.

          If you have a bone to pick on this then kindly do so on your own web site with links back.
          dahowlett
          • What is your agenda Dennis?

            You use ZDNet, largely a sell-side (aka "vendor") ad financed CNet property, as the medium to publish a very brief mention of ZL v Gartner as a segue into an extended rant attacking Gartner credibility (on the basis of vendor revenue-induced bias no less) and suggesting rampant dishonesty throughout the industry. You then go on to use the opportunity to mention (dare I say promote) yourself and your new venture which, not surprisingly, would compete at some level with Gartner. Lovely.

            If I have any agenda at all, it is to highlight hypocrisy at its source, not on another website conveniently distanced from the discussion like a free speech zone at a political rally. Why blog in an open forum if you're not prepared to hear opinions that uncomfortably differ from your own?

            As an aside...since you seem to be confused about the relationship with the buy and sell side of business...

            Buyers (your 'end users') are funded entirely by their organizations' sellers (the folks in sales and marketing that they love to hate). Like it or not, their agendas are inextricably linked.

            Buyers within an organization conveniently choose to overlook that important fact.

            They claim to seek honesty, unbiased opinion and facts so long as none of it applies to or harms their own gravy train. It is highly unlikely that you will ever find a DBA within Oracle, or a developer within Microsoft, or an IT manager within Ford who would publicly denounce favorable opinions of their current employers [written by anyone] even if they believe the opinions to be misguided or misleading. Beyond the extremely rare 'whistle blower', they are not out there on the Internet setting the record straight.

            Yet they have the audacity to demand greater scrutiny, transparency and honesty from the very same machinery (sales, marketing and the media) of other organizations including their competitors. Heck, they have the same expectations about political machinery not in alignment with their own interests.

            It is only when their own organizations, products, services, beliefs and values come under fire that they cry foul.

            Nothing less than hypocrisy. Tell me I am wrong, Dennis. Go ahead.
            josephmartins
          • Gosh

            I guess the point Dennis is trying to make that when your PRODUCT is selling an OPINION you tend to BLOVIATE for the HIGHEST bidder. No?

            If the analyst firm in question were beyond reproach they would be transparently publishing their books. They would also look for FACT and provide FACTUAL Analysis, not OVERRATED OPINIONS.

            JP
            zlfan
          • Parting thought.....

            JP,

            Tend to?

            If I have learned anything about human nature over the past few years, I have learned that psychological projection is rampant in punditry. They 'tend to' project their own undesirable traits onto others.

            Regarding your understanding of facts and opinions, I respectfully suggest that you read my comments in the FTC thread on StorageMonkeys.

            [Note: StorageMonkeys is not my website. I'm just a free member who has contributed extensively on this subject in recent weeks.]

            Although I believe all of my comments are relevant to Dennis's original post, I have decide to write a more detailed response elsewhere. I'll publish the link on Twitter when it is finished.

            And Dennis, I do hope you'll find the time to read it. Our disagreement here aside, your thoughts would be appreciated.
            josephmartins
          • Look in the mirror

            Isn't Dennis' product also an opinion? According to your analysis, with which I can't find any fault, that would mean he would tend to bloviate for the highest bidder.

            Let me replace references to Gartner in your comment with references to Dennis:
            [i]
            I guess the point [b]JoeNemo[/b] is trying to make that when [b]Dennis'[/b] PRODUCT is selling an OPINION [b]Dennis[/b] tends to BLOVIATE for the HIGHEST bidder. No?

            If [b]Dennis[/b] were beyond reproach he would be transparently publishing his books. He would also look for FACT and provide FACTUAL Analysis, not OVERRATED OPINIONS.
            [/i]
            JoeNemo
  • RE: Gartner in the dock over Magic Quadrant

    The Magic Quadrant is a special case of what is called a multi-attribute decision model (MADM) that seeks to quantify qualitative things. The problem with the MQ that I have voiced to Gartner several times is that the criteria they use are HIDDEN and VAGUE. If you want people to be able to improve, you need to provide more information about how you are rating them and why. Gartner does not do this, and they argues they are being scientific. This is why I prefer Forrester's Wave and Datamonitor's Decision Matrix. At least they are explicit about their criteria so that you can see what things might need to change to get a different rating.

    If you want a good example of how politically motivated the MQ is, look at Microsoft's rating in the leaders quadrant in their most recent MQ on enterprise content management for SharePoint. While SharePoint has it's place in an enterprise, it is poor at most functions that most people would call content management. BUT, it is broadly deployed and has the MS name behind it.
    smprezbo
    • Keeping it simple

      How about some of the transparency of which ZL speaks?
      dahowlett
    • Science

      And as anyone that has done some engineering an scientific research will tell you, peer review is a part of the process. Your data, your calculations and your conclusions have to match. Otherwise it is just ... not scientific.
      zoe000@...
  • RE: Gartner in the dock over Magic Quadrant

    It certainly was a huge coincidence that shortly after Informix decided not to pay the big freight to Gartner, their previously wonderful products were suddenly panned in the MQ. Informix stock took a nosedive and then the company was sold to IBM. Many of us in the industry viewed that as direct cause and effect.
    mmays@...
    • Evidence?

      Do you have recorded evidence that can be exposed? If so then I'm sure all readers would be interested.
      dahowlett
      • Analyst

        just like we need to force our financial analysts to explain how they get to a conclusion on a stock, the same should go for "Gartners" of the world. Otherwise, what prevents them from doing a pump-and-dump. One MQ you are on top. Sell your stock. Run away. Has anyone looked at Gartner analysts stock buy and sell patterns along with their analyst opinion???
        zoe000@...