A paper on aggregation method design for prediction markets
Predicting the Future
Kay-Yut Chen, Leslie R. Fine and Bernardo A.
Huberman
HP Laboratories, Palo Alto, CA 94304, USA
Abstract. We present a novel methodology for predicting future outcomes
that uses small numbers of individuals participating in an imperfect
information market. By determining their risk attitudes and performing
a nonlinear aggregation of their predictions, we are able to assess the
probability of the future outcome of an uncertain event and compare it
to both the objective probability of its occurrence and the performance
of the market as a whole. Experiments show that this nonlinear
aggregation mechanism vastly outperforms both the imperfect market and
the best of the participants.We then extend the mechanism to prove
robust in the presence of public information.
So if we follow the recommendations of this paper, the mathematics of
which is beyond my level of comprehension, we should set up a two-stage
method:
Stage 1: Run an information market to extract risk attitudes
from the participants,as well as their ability at predicting a given outcome. Assign
a value for each participant's risk attitude (1 is risk neutral, >1 is risk
averse, <1 is risk loving).
Stage 2: Ask individuals to provide forecasts about an
uncertain event, and reward them according the accuracy of their forecasts.
Aggregate the individual forecasts to predict the outcome.
·
Use the risk attitude value discovered in Stage
1 in the aggregation method to account for how participants use their private information.
·
Identify public information within the group and
subtract it in the aggregation method so that when the same information
is used by multiple participants to make a prediction it doesn’t get counted multiple times.
Analysis by Art Hutchinson (link)
My take: While such accuracy may be essential for some
applications (e.g., forecasting demand for a well-established product),
it appears to require significant time and expertise to do right. (The
math is dense, to say the least.) An even more important implication -
especially for resilient strategic planning - is that it appears to
lack a dynamic component: a way for a group to continuously
seek, refine and assimilate new information about a complex and
changing market or competitive environment. Finally, it seems to rely
on a closed system, where 'good forecasters' are identified up-front on
an assumption that their forecasting ability is generic and not likely
to be found in otheres. Not enabling new, marginal participants with
correct (but possibly 'heretical') information to enter and influence
the process can be deadly. In other words, the approach, while clever
and useful for static applications, appears to lack precisely the
openness and adaptability essential for anticipating the emerging
likelihood and potential impact of discontinuous change, (i.e., sudden, unprecedented surprises). Being able to quickly see inflection points
developing in a measure of conventional wisdom is often more important
than achieving the last decimal point of forecast probability.
