Fascinating. The set up:
At Google, employees are encouraged to go online and place bets on a prediction market — an exchange that tries to forecast events based on the money wagered on a particular outcome.
Prediction markets have been used for years to predict things like elections. At Google, they are used, of course, for business. In the last two and a half years, 1,463 employees have made wagers with play money (Goobles, as in rubles) on questions like: will Google open a Russia office? will Apple release an Intel-based Mac? how many users will Gmail have at the end of the quarter?
The pay off:
According to the report, “Using Prediction Markets to Track Information Flows: Evidence From Google,” which was presented Friday at the American Economic Association meeting in New Orleans, the strongest correlation in betting was found among people who sat very close to one another, trumping even friendship or other close social ties.
This is tangible evidence, the authors argue, that information is shared most easily and effectively among office neighbors, even at an Internet company where instant messaging and e-mail are generally preferred to face-to-face discussion.
Link to the report (PDF download)
John, in many situations prediction market is a useful tool for information gathering, but in many situation it may cause conflict of interests. Prediction markers give useful information if the participants take them seriously. But if the same particpants are also the decision makers then it causes serious conflict of interest.
Here is an example. Suppose you were a major supporter of Bush in the last presedential election. You wanted to help him out but the law puts an upper bound on the help you could offer him. Law may also disallow you to buy votes for Bush. Well, you travel to Florida. Take a position that Bush will lose with 2000 Floridians (you can do so with very little money). That is these Floridians take the position that Bush will win. Comes the election day, these 2000 Floridians get some money from you if Bush wins, so they vote for Bush.
That way you get your true wish! But effectively you bought 2000 voters.
Kamal,
That makes no sense. How exactly are you going to round up 2,000 Floridians to have a bet with? Furthermore how much are you suggesting it costs to buy their votes? A NYT article recently quoted research(http://freakonomics.blogs.nytimes.com/2007/11/16/is-it-smarter-to-sell-your-vote-or-to-cast-it/) that 20% of college students would sell their vote for an iPod Touch (retailing at $299). So it would cost this innumerate billionaire $600,000 and he still wouldn’t have any guarantee that his 2,000 Floridians actually bothered to go and vote!
There’s no need to buy anyone an iTouch; just spend ~$0.12/click for hundreds of thousands of click-throughs to a blog explaining the virtues of your candidate, ideally CEO-targeted to the swing states. I’ve done that in both U.S. & French presidential elections.
Nigel, it will cost very little.
Suppose the odds of winning/losing Bush were close to 50% before the election. John will run a betting service on the internet. If the IP address turns out to be from Florida, then odds are offered that a rational Floridian gains in expected sense (for an example 40/60 odds).
John can hedge this money by making the inverse bets if the IP address is from a Republican state. So even though John makes 2000 bets in Florida, most of them are hedged.
You can also see that within a corporation this conflict of interest could also give optimism in prediction market. It has both good and bad. Bad because you are not getting the true information. You are getting a more optimistic view of the world than it actually is. Good because now the employees, if they take their bets seriously, are even more motivated to make the optimistic picture come true!
I would like to echo to a degree what Kamal is saying. Not necessarily about conflict of interest, but about the corollary: whether or not this is truly “wisdom of crowd”.
As far as I understand it, prediction markets are based on the Surowiecki “Wisdom of Crowds” ideas.
And as far as I understand it, in order for Wisdom of Crowds to work, it requires four pre-conditions: (1) diversity of opinion, (2) independence of decision-makers, (3) decentralization, and (4) an opinion aggregation mechanism.
I’m sure Google has condition #4. And maybe #1. But one big question that this study raises in my mind is whether the participants in this prediction market arrive at their conclusions independently of each other (condition #2), and whether they are decentralized in their ability to draw upon separate knowledge bases in order to arrive at their conclusions (condition #3).
It seems to me that if “the strongest correlation in betting was found among people who sat very close to one another”, then that means market behavior is arrived at non-independently. Co-workers, sitting near each other, dependently influence each other and arrive at the same conclusions.
That fact alone destroys the foundational basis of wisdom of crowds, and therefore of prediction markets.
This is not quite the “conflict of interest” that Kamal is talking about, but it shares with his observations the fact that things which should be independent are not.
So what is interesting to me about this study is that it seems to call into question the whole idea of wisdom of crowds. Thank you, Google, for publishing it.
JG you are right that there is a connection to Surowiecki’s “wisdom of crowds,” but don’t take Surowiecki’s four points as absolutes. While independence among market participants would likely improve outcomes, the modest lack of independence found (correlations in trading patterns among people who shared and office together, among people with the same non-English native language, among people with other shared professional associations) wasn’t sufficient to undermine the relative accuracy of the market predictions.
The authors don’t really emphasize the accuracy of the predictions in the paper, but some sense of how accurate the markets are can be inferred from the discussion of bias. Also of note is that the biases detected declined over the sample period, meaning the markets became better predictors over time.
Folks interested in prediction markets can check out the Midas Oracle group blog on the subject, among other online resources.
Hmm, ok. Thanks Mike; I will re-read the paper with your points in mind.
And I know I am critical of Google much of the time in my comments. But I was serious; I do thank Google for publishing that paper. Google research does so very little publishing, so very little giving back to the “open source” of ideas that paper publication represents, that it’s nice when it does happen.