Prediction Markets Foretell Online Trends
By Tom Ryan
A retail-industry think tank at the University of California, Riverside, has launched a new website that relies on “prediction markets” to
foretell online sales and other internet-related trends, such as which e-commerce
sites will likely survive.
Under the method, also known as “competitive forecasting,” websites pit users against each other to determine who is the most prescient about a certain topic. Proponents claim the method, publicized in James Surowiecki’s book, The Wisdom of Crowds,
can yields more accurate results because participants care much more about
their answers than in a phone survey.
“It’s increasingly hard to get people to participate in research studies,
but they’re very eager to participate in prediction markets,” Thomas Novak,
a director of the Sloan Center, the think tank, told The New York Times. “Consumers
are actually coming to you, which seems to me what the web is all about.”
Among
the best-known successes of the prediction markets is Cantor Fitzgerald’s Hollywood
Stock Exchange, which last year predicted 32 of the 36 major-category
Oscar nominees and over the last three years has correctly predicted 92 percent
of the major-category Oscar winners. The Iowa
Electronic Markets
often predicts election results more accurately than pre-election polls.
Under Sloan Center’s eLab
Exchange, users sign up for free, assume an online
identity, and then answer a slate of questions. The site ranks users on the
quality of their answers, and offers a gift certificate of $25 monthly to the
top-ranked participant. Every three months, the top-ranked user wins a $500
gift certificate.
Prediction markets offer unreliable results if too few people care enough about a topic to answer questions carefully, but Mr. Novak believes $25 and the promise of e-commerce industry fame will attract at least “hundreds of people.” In addition, the site was designed so people could choose the most interesting questions to answer. The eLab Exchange will count on discussion boards and blogs devoted to niche internet topics to drive traffic.
While the Sloan Center doesn’t intend to market its data, other companies, particularly software vendors, are coming up with business research practices around prediction markets. According to the Times, these companies say demand has jumped over the last year, both from organizations like the Sloan Center who want to create public sites, and from corporations like Best Buy, Hewlett-Packard and others looking to create private websites to devise business strategy with the help of their employees.
Zubin Dowlaty, vice president of emerging technologies for InterContinental Hotels Group, recently used online prediction markets to ask employees which new technologies the company should invest in.
“Compared to polling people, this significantly improves the process of harvesting and prioritizing ideas,” Mr. Dowlaty told the Times. “And the cost is extremely low, relative to surveys or engaging a consulting firm.”
But Robin Hanson, an economics professor at George Mason University and a consultant to companies building prediction markets, said he was skeptical about many businesses forming around the concept, because outside of politics, movies or sports, few subjects entice people to render carefully considered opinions. “It’s kind of like a job,” he said.
Discussion Question: What do you think of the value of prediction markets around consumer research? In what ways might they be better than traditional consumer surveys? Can participants be enticed to express opinions about consumer products, as they do with movies and sports?
Mr. Whalin is right to be candid about most business forecasting: it is indeed an inexact science, and many predictive research methods are inaccurate. Still, nobody disagrees that it is smart to produce the most accurate possible forecasts and predictions. And that is why prediction markets are so exciting; their accuracy relative to the alternatives is compelling. In fact, they are often useful for illuminating the flaws in competing forecasting methods.
Yes, every year, some “experts” are more wrong than others–and some are more right than others. But aggregating well-informed individuals’ collective knowledge produces relatively more accurate forecasts than even the best individual forecasts.
Here is a simple thought exercise to illustrate this concept: If 30 people are asked to estimate the number of jelly beans in a jar, the average of their estimates will very likely be more accurate than the single most accurate estimate. Far more interestingly, the same is true if you’re forecasting next year’s holiday retail sales.
And prediction markets are an even more elegant and efficient aggregation tool than averaging. They reward good information and penalize bad information. And because the accountability is cumulative, the longer they’re operated, the more accurate the forecasts become. If the NRF guessed wrong in 2006, it will have less in its play-money portfolio to wager on its 2007 forecast.
As prediction markets’ relative accuracy becomes more widely understood and accepted, individual experts’ (and institutions’) value as forecasters will be diminished.
Many who experiment with them will fail, but those who understand where and how to apply them will have better information than those who do not.
It’s difficult to extrapolate the potential value to retailers of this methodology for market research. At a macro level, particularly with gaging overall spend potential for “seasons” such as BTS or Holiday, this may have significant potential, especially in the model described in the article. It seems to me, though, that the value of the prediction is a function of the availability of worthwhile information and the degree to which the participants are informed. I know that the theory assumes the profit potential or competitive aspect of the model will incent participants to be as well informed as possible.
How would this potentially be applied? WM, as recently as yesterday, has an acknowledged issue in home fashion and apparel. Could this process be used to predict which price points, potential designer names, or categories people might respond positively to? I can’t really see that.
What about for grocery, in terms of expanding or contracting merchandise categories, adding RTE meals, etc? Could this be used to predict the types of products to concentrate on? Yes, perhaps that might work.
Overall, I’m not seeing the direct and immediate application.
James Surowiecki’s book, “The Wisdom of Crowds,” is a perceptive book that I recommend to CPG managers for a better insight on why traditional means of surveys are not as effective today, as they once were. I agree that online prediction surveys get a more passionate and committed response than the traditional telephone survey, however, I think we should take with a grain of salt any supposed reliable results because passionate opinions might also be the not-so-silent minority, and it might not truly reflect the right predictable numbers where consumers and retail are concerned. Nonetheless, it’s very interesting!
Motley Fool CAPS and Meritocracy are two examples of stock market investment games, run online, with play money portfolios. Meritocracy even has a mutual fund, run with real money, based on participant traders with the best track records. It’s clear that making real money investment decisions based on the best players’ track records for both sites, is well worthwhile. It’s also clear that ratings on Amazon are often great indicators of the best and weakest products. I suspect that aggregating Amazon ratings by brand might be an excellent guide for investors as well as retailers of the appropriate categories. For example, an appliance store would know which brands and models in a given assortment would be more likely to raise customer satisfaction.
Over the course of nearly 20 years, prediction markets have proven themselves more accurate than many other forecasting methods–especially polls. The reason? People with good information have an incentive to share it, and people with none have a disincentive to guess.
The key to these markets’ success depends on a few critical factors: liquidity, diversity among traders, careful selection and wording of the “contracts” (questions) traded on the market, and adequate incentives to participate. Traders must be rewarded for a) being right b) before others. They must have sufficiently broad and different information in order for the market to aggregate it meaningfully. And the questions they are attempting to answer must be answerable.
The operators of the eLab Exchange should not be too concerned about Robin Hanson’s comments, which apply more directly to businesses seeking to generate revenue from the operation of prediction markets. In the U.S., where online gambling is outlawed, very few play-money prediction markets are operated as successful businesses. (Abroad, prediction market operators make money by charging traders transaction fees.) But a market operated by a think tank or industry association and funded by dues or sponsorship has promise.
Just a note for those who are interested in exploring these tools further: prediction markets and “competitive forecasting” are two different things. As I understand it, the eLab Exchange is actually not a prediction market.
Prediction markets, as their name implies, rely on a market mechanism to produce probabilistic predictions and forecasts. The price of a given “contract” implies the relative probability of a certain outcome, be it binary (e.g. will Lee Scott retire before 1/1/2008–yes or no?) or a value (e.g. what will Wal-Mart’s retail sales be for fiscal 2008?)
Competitive forecasting, pioneered by Emile Servan-Schreiber of Newsfutures, uses scoring rules to rate forecasts but does not require participants to translate their forecasts into trading decisions.
For more information, I suggest these two excellent resources:
http://www.chrisfmasse.com
http://www.midasoracle.org
Nonsense! For years researchers have been asking consumers what they are GOING to buy only to have the consumer buy something else when they are actually in the store. Predictive research has some benefits, but it will never take the place of actually observing what customers do when they are in stores.
Those of us who spend our lives trying to determine what consumers are going to do next make predictions every year about how many percentage points holiday sales will increase this year over last. And every year some of us are wrong and some of us are right. These predictions come from the NRF, ICSC and economists with the nation’s best known consulting, brokerage and research firms. It is an inexact science that ultimately requires an informed guess that can be wrong or right. We only hope we are right more than wrong.
Considering the time-worn, time-proven adage that in the supermarket, 70% of purchase decisions are made at the shelf, one wonders how much thought shoppers devote to their responses to these–or any–surveys. This dynamic is further complicated by the tendency for online opinion-givers to complain and respond negatively than to be supportive and constructive. Just read any online forum. Perhaps the results of these surveys should be trusted more for what respondents dislike rather than for what they like.
Consumers can be enticed to comment on anything. Especially when there are cash incentives involved along with the promise of fame and the ridiculous amount of respect that alleged celebrity may bring. But having a self-selecting audience and inviting members to assume an alias (or several) cannot seriously be expected to predict anything either sensible or remotely reliable.
I am fascinated by this topic, having never heard of it before. I know I personally avoid responding to phone surveys I receive at home, and yet offering an opinion which raises my visibility (like I do with RetailWire for instance) is something I am very happy to do.
I think next I’ll visit the featured website and see what is going on there.
In my opinion, potential clients of these prediction marketers could save themselves a bunch of time and money. There exists today a product which can provide the same level of statistical accuracy.
The data is easily accessible, easy to understand and requires only that you input the question or concept for which you are interested in predictive output. It is widely available and very reasonably priced, for a one time investment of under $10.00 you will be able to instantly access the predictions any time and anywhere. Let me introduce you to the Mattel Magic 8-Ball.