Trials and tribulations of product testing

When it comes to product testing, it seems market leaders have more to protect.

It takes a lot of work to bring you our product comparison reviews each and every month. From beginning to end, I can tell you in no uncertain terms, it is a pain in the bum.

But it does give us an interesting insight into how technology vendors operate. We at T&B endeavour to bring you reviews of products that we think you will be purchasing. Vendors endeavour to provide us with products that they want you to buy. Often those two objectives meet, sometimes they don't. When the latter occurs, what happens next is a struggle between us and the company marketing executives.

Sometimes it is a matter of timing. For example, vendor X would love to submit a product for review, however, it is releasing a newer model in a couple of months and doesn't want to push a soon-to-be old product.

Sometimes the company will be honest and say "we don't want to push product Y right now, we would rather customers were buying product Z". In our guide to archival storage, we invited vendors to submit up to three products each, one for each category: disk, tape, and optical. By and large, most vendors decided to only send in one product: disk. No prizes for guessing what product brings the most revenue.

But the attitude that really gets me, and one that I have run into a couple of times in the past few months, is what I will call "big-vendor syndrome". They are the vendors with the largest market share in their product category, and therefore don't feel the need to put their products up for comparison with their competitors -- why take the risk that you may end up looking bad, when frankly, you can afford to lose out on a bit of coverage in a magazine.

That was one of the worst, and most blatant, attempts I had seen to control the press.

Cisco, for example, won't provide products for comparative reviews because it has a company in the US that benchmarks its products. Hmmmm, only one company is allowed to test its products . . . sounds like a good way to keep a consistent market story.

Another leading vendor caused us some grief recently. Its initial response to our invitation to submit a product for review was of the "why-should-we-bother-when-we-own-the-highest-market-share" variety but for some reason, the company changed its mind and decided to participate.

Nervous about handing over such an expensive product for review, the company wanted us to sign a contract stipulating strict licence terms. Embedded in the document was a clause stating that it would have the rights to approve any testing results before publication, and not only of its own product, but of all the other vendors' products that were also tested.

That was one of the worst, and most blatant, attempts I had seen to control the press. Is it just a standard marketing attempt to own the message, or is this company afraid that its market-leading products are perhaps not up to scratch?

Either way, it was wishful thinking -- and never mind the fact that it handed over this eight-page legal document two days before testing was due to start. A matter of bad timing, or a way of sneaking that nasty clause through?

Vendors make many attempts to control the messages you receive about their products, and during the preparation of our product reviews, we get to witness a lot of them. So if you ever notice the absence of a leading company in our reviews, it probably isn't from a lack of trying on our part. What you should probably ask is, why didn't it want to participate?

Natalie Hambly is editor of Technology & Business. Send feedback to

This article was first published in Technology & Business magazine.
Click here for subscription information.