Directed fantasy - market forecasts

My colleague Larry Dignan posted Tablets will displace PC units; Time to pick winners and losers yesterday and although the commentary on Gartner's research was interesting, I wanted to take it in a different direction. I'd like to comment on market research, forecasts and how they're both directed forms of imagination and fantasy.

The process of gathering market data is fairly straightforward, but is very time consuming and staff intensive. Data is gathered. It is compared to historical data. Tables showing how revenues or shipments have changed over time is then published. The next step, the forecast, is based upon the historical data in those tables and a whole bunch of other things, things the reader may not find acceptable or useful.

Lies, damn lies and statistics

When market research firms, such as International Data Corporation (IDC) or Gartner Group publish a report containing historical data and a forecast, it is the result of a great deal of hard work. They're presenting facts as they have been able to collect them and the result of their analysis, their insight and their opinions. They are not, however presenting truth. They are presenting a collection of facts and their interpretation of those facts.

In essence, the research firm has stepped from the concrete "we looked at the world this way and counted this number" to the world of directed fantasy.

Data collection

Key questions to ask when considering a report offered by a research firm are:

• What is being counted? That is, what was included and why.
• What was excluded? That is, what wasn't counted and why.
• How was it counted?
• Are unlike things, such as product revenues and product shipments, being counted together?

Some research firms "cherry pick" only the top suppliers in a market and make no attempt to dig deeper to find smaller firms or firms in other geographical regions that offer similar products or technology. This approach might be useful for some purposes and totally unsuitable for others.

Data analysis

Some research firms have an extensive segmentation (some times called a taxonomy, a framework or a codex) of the market that is exhaustive and has mutually exclusive categories. They go to great efforts to make it impossible to double count revenues or shipments or to count these things in the wrong place.

The key questions here are:

• Does the segmentation make sense?
• Does the segmentation represent reality or are apples being counted with strawberries and the results called apples?
• Are revenues and shipments being intermingled?

Data presentation

How the data is presented may make accurate data unusable.  Graphs containing no totals and unmarked axises don't really communicate anything useful, but often are presented as if they were critical pieces of insight.

Don't be influenced by a beautifully presented piece of fluff.

Data forecasting

The jump from historical data to a forecast often hides quite a number of steps.  Looking at the changes in revenue or shipments over a specific period of time is pretty straightforward. How the forecast was created is murky at best.

Forecasts are often the combination of the historical trend lines, a series of assumptions about what is going to happen next, another series of assumptions about how each previous assumption will impact the total and, finally, the judgment of one or more analysts. Any step in this process may make the result more or less useful for any given purpose.

The key question here are:

• Is the historical data accurate?
• Is it based on a reasonable segmentation or framework?
• Was it counted properly?
• What assumptions were used to create the forecast?
• What weight was each of the assumptions given?
• What impact did analyst opinion and insight have on the final result?

If any of these don't ring true the research isn't going to be useful.

Summary

Forecasts are a bit like sausage. Those who are wise find out what basic ingredients have been included, how they have been processed and what spices have been added. Sausage that is a nice addition to breakfast may not be robust enough for use in another meal.

Forecasts should be seen as a tool not as infallible gospel.