Over at Paul Mooney's blog, he's shared the contents of an e-mail from a Notes customer. The customer describes their experience with Microsoft and the Application Analzyer for Lotus Notes:
had to setup a workstation to run the analyser and before Microsoft came in I ran the analyser on all our apps (approx 500 and my results were 100% completely different than the results Microsoft presented. I had approx 200 apps in quad 3 and approx 300 apps in quad 4.Actually, I have a proposal, which I haven't vetted with IBMers but suspect I could make happen.
Microsoft had all the apps in Quad 1 and Quad 2, mostly in Quad 1.
We have all custom apps created since notes 4, we don't use standard templates and at least 300 have workflow in them.
I am basically hung out to dry here. How do I compete with the reporting Microsoft did?
What if IBM, or an IBM partner, offered to come in and run the tool for this customer? If indeed the results being presented are different than the "out of the box" Analyzer results, at a minimum it will raise questions. It may be that the reporting tool needs some tuning to run accurately in a customer environment, but if Microsoft's results are so different, the rest of the discussion get challenged.
The sad news here is we have an exact scenario that both Paul and I suspected would play out: a customizable tool gets different results depending on who runs them. This from the vendor who wants to be the customer's partner for the future.