Gartner in the dock over Magic Quadrant

Friday October 23rd will see Gartner argue a motion to dismiss a complaint by ZL Technologies Inc about the famed Gartner Magic Quadrant. According to court papers, Gartner will argue to dismiss based on First Amendment rights citing that the Magic Quadrant is not meant to represent statements of fact but is based on pure opinion.

Friday October 23rd will see Gartner argue a motion to dismiss a complaint by ZL Technologies Inc about the famed Gartner Magic Quadrant. According to court papers, Gartner will argue to dismiss based on First Amendment rights citing that the Magic Quadrant is not meant to represent statements of fact but is based on pure opinion.

I'm not qualified to argue the legal ins and out of the case and so if anything I say here veers in that direction then you can safely ignore it. However, ZL's plaint is not really about constitutional rights but about what it sees as an abuse of market power.

In a statement sent to me, ZL says of its original plaint:

ZL claims that Gartner’s use of their proprietary “Magic Quadrant” is misleading and favors large vendors with large sales and marketing budgets over smaller innovators such as ZL that have developed higher performing products. The complaint alleges: defamation; trade libel; false advertising; unfair competition; and negligent interference with prospective economic advantage.

This is a topic that from time to time gets aired by those who believe Gartner (and other industry analysts) are in what some regard as an incestuous 'pay-to-play' arrangement with vendors. Earlier in the month, Gartner analyst Tom Bittman hit out at those who argue 'pay to play:'

As an analyst at Gartner, I can’t describe how angry I get when I read bloggers spouting as “fact” their opinion that I and my teammates have no integrity. That we can be “bought.”

In my 14+ years at Gartner, I have never, ever allowed a vendor to influence my opinion with anything but facts. Period. They have certainly tried to influence me with non-facts. I can say this definitively – it has never worked.

I don’t think there is a single vendor that I have dealt with who has not been very angry with me at some point. Tough. I’ve been yelled at by many IT executives – including the CEOs of Microsoft and HP, and many other firms. I can’t think of one of those cases when I changed my analysis one bit. I can’t speak for other firms, but at Gartner, getting yelled at by a CEO is a badge of honor. Being proven right as time goes on – priceless.

I certainly spend time helping vendors with their strategies and their marketing messages – and I enjoy doing it. Frankly, the ones who yell at us the most seem to respect our opinion the most. We can spot holes a mile away, and engaged early enough, we can help vendors fill those holes with real product offerings – that not only help the vendors, but help our end user clients. And my primary business is helping end users.

The post attracted 33 comments including several from Vinnie Mirchandani and myself. Vinnie argues pretty much the same thing that ZL is saying (you need to check the post as there are no comment links):

Tom, as for your former colleague and now a blogger I guess I see both sides. If you think it is just bloggers raising the issues, you are not reading the market, including your customers well.

The issue is not at an individual analyst level. It is at the firm level. You know it and I know it that vendor sourced revenues have been growing nicely as percentage of Gartner revenues. You and I know the majority of top 25 Gartner revenues are vendors like IBM – and individually they dwarf what even the biggest user entities like GE or BP pay you.

It would help if Gartner was transparent about that, rather than avoid talking about it, and if Gartner aggressively showcased how those vendor interests and subtle and not so subtle pressure from them is balanced with user inteersts. The answer usually is the Gartner Ombudsman blog does that. Seriously?

The other thing is 2 of Gartner’s best known tools – the MQ and the Hype Cycle are skewed towards larger vendors and against startups. The MQ rewards market share and viability or tenure in a market. By its very definition the Hype cycle warns people about newer stuff. So Gartner comes across as pro-establishment with more skew towards larger, incumbent vendors

In the field I see time and time again vendors using your metrics where it suits them. I see economics that are dated. In a consulting assignment last year, the CIO asked me fairly loudly during a presentation when a vendor put some of your metrics “so is that 30 or 40% over priced?”

[my emphasis added]

Tom responds by arguing that the MQ is more scientific these days:

In your day, MQs were developed by smart analysts placing dots. Today, we have a much more rigorous mathematical model – we don’t just place dots. It isn’t perfect, but it’s better. “Ability to execute” includes viability, which will tend to favor large, incumbent vendors. But “vision” actually favors the start-up. When we execute the MQ well, you clearly see the visionaries, the incumbents, etc. It is in our best interest to show that variety in an MQ.

Just because something is more scientific doesn't make it fact as any disinterested observer of the climate change debate will tell you. But there is no denying that Gartner opinion stated as fact has become a powerful tool for vendors to wave in front of customers. I argued that:

@tom: I get complaints pretty much every week of the ‘pay to play’ argument so whether you believe it or not is immaterial. It goes back to what @vinnie says about firm level issues and the corporate emphasis on aggressive selling – or as one of your major clients puts it to me: tin cupping.

I appreciate and respect that individual analysts try maintain their independence and I see your examples. I know several of your specialist analysts in my specialist space and I would not doubt their personal integrity.

It's worse. In one conversation last week, a senior firm's representative went on to add: 'And you know what, it doesn't stop at tin cupping. They double dip. They say they'll help you with strategy, charge you for the pleasure and then go on to sell analysis based on that to say exactly the opposite.'

In another comment, Jonathan Yarmis was far more blunt:

I’ve been on all sides of this argument (Gartner analyst, non-Gartner analyst, AR professional) and to argue that vendor money does not influence perceptions and outcomes is hugely naive. My standard spiel as an AR practitioner was that money did not necessarily equate to influence. I’ve seen people spend a lot of money to no apparent benefit. I’ve seen people spend no money and get great outcomes. But money well spent equates to more positive outcomes.

If you think all that money vendors are spending is purely because of your insights, again you’re being naive. There’s an influence component and an insight component. Why do you think they spend so much more with Gartner? Are your insights really that much keener? Or are the vendors more concerned with your market reach and therefore the need to more strongly influence you compared with everyone else?

So, the vendors are spending because they believe they’re influencing you. You’re claiming you can’t be influenced by this. Either you’re wrong, and there is an influence element at play, or they’re wrong and they shouldn’t be spending nearly as much money with you. Which is it?

It is for these and other vendor related issues that Vinnie, Ray Wang, Frank Scavo, Oliver Marks and I decided to establish Enterprise Advocates. For too long the voice of the customer has been sidelined in favor of market driven analysis that in our view is massively skewed towards the 'establishment.' The vendor pieces may move around on the chessboard of enterprise computing but the established names are pretty much always there. Go to any trade show and you'll see hordes of analysts being shuffled off into side rooms for 'exclusive' briefings, sometimes under NDA, sometimes not.

To the credit of a handful of vendors, they have chosen to embrace the independent blogger/analyst voice but there is always pressure upon us to tone down. It goes with the turf. SAP has the best developed blogger program of any software vendor. That's one reason why you hear so much about them in this blog. That doesn't prevent me getting the late night calls asking why I said this or that. It sure as heck doesn't prevent at least some executives trying to tear us a new one when we bang on about maintenance.

Others try and ignore us but what they don't realize is we're not going away. Our clients demand that independence. It's what they pay for. Simply trying to shut us down is counterproductive. That doesn't stop the established hand wavers from boosting technologies but that's OK too. Practical experience and direct customer experience wins over marketing when it comes to signing the check. That's perhaps where ZL would be best diverting its legal spend.

Even so, ZL points up some uncomfortable assertions. Even if it lucks out in court it will have been successful in drawing attention to important issues. As the ZL statement said, it hopes to highlight:

- Fair Disclosure on Conflicts of Interest – Gartner generates its revenues from payments made by the same vendors whose products it evaluates. Similar to the new rules now being imposed on financial ratings agencies on Wall Street, Gartner should be required to disclose the revenues received from the vendors it ranks. - Fair Disclosure on Evaluation Scores The tech industry would benefit if Gartner were required to disclose more data in its evaluation process and disclose component scores so vendors know exactly where they are lacking and by how much and take corrective action. Currently, there is zero disclosure, which can lead to arbitrary placement, with no recourse and no basis for appeal. - Better OversightGartner currently has an employee act as ombudsman to handle disagreements. The conflict of interest is self-evident in the way ZL’s concerns were summarily dismissed with little supporting evidence. There is a crying need to establish an impartial ombudsman similar to those found in public media, in order to ensure purchasers that they are receiving impartial analysis.

These are all points I welcome.

I spend a lot of time talking to other so-called analysts. Almost to a man/woman (but not entirely) they are 70-80% in the pay of the vendor community. Many believe they are independent. However when I ask who really pays the bills they go silent. Shout all you like about defending your independence but at the end of the day? Show me the money.

Would I work with a vendor? Of course. It's inevitable that vendors need the buy side and where that requires a consulting component then there needs to be a transaction. But it is always on the strict understanding that what I am saying is from the buyers' perspective and that I am not here to defend anyone's marketing. It also means I will strictly limit the amount of paid for assignments coming from that side of the 'house.' Oh yes - and any of those assignments will be disclosed. Promise.