Air Time: Tech Market Research

I have an observation about the effectiveness of meteorologists in upstate New York: Their accuracy drops by about 50 percent each day into their long-term forecasts. In a perversion of this realization, I find myself hoping the Monday forecast calls for rain the following weekend, as it instills confidence in a nice weekend at the lake.

My jaded perspective on weather forecasting similarly infects my opinion of industry market projections, which often do far more harm than good by creating a false sense of reality and stimulating investment in bad ideas. I've come to trust market research experts' ability to tell me who sold what for how much last year and, to a lesser extent, who will sell what for how much in the coming year. After that, it could be sun--or it could be rain.

Your TV meteorologists, if they're any good, know just how tenuous those long-term forecasts are, but proclaim them anyway, because the boss wants eyeballs. Although largely irrational, people are attracted to long-term forecasts because they're the most tangible contemporary effort to predict events that affect us all.Forecasts from the IT market research community also attract eyeballs. Savvy tech analysts know how quickly markets change. Disruptive technologies wreak havoc on simplistic market projections, especially long-term forecasts. Still, the consumer appeal of these forecasts is so high that it's not uncommon for a journalist or analyst to insert forecasts into a technology assessment, in an effort to create buzz.

In most cases, long-term market forecasts are an abuse of the spreadsheet's good name, simple models constructed on simple assumptions about a limited number of variables. These underspecified models usually do more to mislead than to inform, distilling complex market dynamics into snazzy infographics. Unfortunately, making the models more complex doesn't necessarily solve the problem. Even if a particularly bright analyst could formulate a fully specified market model, the right model only gets you halfway there. Assumptions also play a role. If a brilliant political scientist could develop the perfect model to predict voting behavior, it wouldn't be worth much if the assumptions about voter turnout are false.

Some might call forecasting more of a black art than a science. What that really means is that people are making up formulas to match their hunches and using numbers--often with decimal points, no less--to provide statistical credibility to best guesses. Equally offensive is the practice of one major tech analyst firm, which uses numerical probabilities to assess the future. Come now, are you sure it's not a .8 probability rather than a .7?

It's reasonable to suppose that you, with your obvious good judgment (based on your choice of information sources), aren't easily duped by long-term market projections. The same may not be said for some bosses, the technology companies with whom you do business, or the venture capitalists and other financiers who fuel--or sometimes stifle--market innovation. They, too, are smart people, but when they see similar long-term market projections from multiple sources, they often start to believe it as truth.

Business managers seeking to use technology as a competitive asset have an insatiable appetite for information that can help them predict the future. Successful businesses and personal fortunes have been made on good guesses about the future, moves that have led many of us to look back and wonder, Why didn't I think of that?To a degree, market forecasts can be self-fulfilling. By predicting a bleak future, they discourage investment, either in development or implementation of technology. Sometimes, those are lost opportunities. More often, they are largely meaningless, background noise that entertains. Place too much trust in these pseudoscientific predictions and you may find yourself walking through a thunderstorm without an umbrella.

Dave Molta is a Network Computing senior technology editor. He is also assistant dean for technology at the School of Information Studies and director of the Center for Emerging Network Technologies at Syracuse University. Write to him at [email protected]