Here are two more profiles of Bueno de Mesquita:
The NYT article provides the most detail I've seen on how Bueno de Mesquita's model actually works. Tetlock is wary of him for being a hedgehog, and concludes Bueno de Mesquita is better than other experts, but that's not saying much.
From an interview, here is Bueno de Mesquita's own recommendations on where to learn more: "Interested readers should read my 2002 book Predicting Politics (Ohio State University Press) or my 1994 book (co-edited with Frans Stokman) European Community Decision Making (Yale University Press) or my 1997 article in International Interactions for explanations of how the model works. They should also read Stanley Feder’s 2002 article in the Annual Review of Political Science for an evaluation of what the model can and cannot do based on the experiences of someone who used it more than a thousand times at the CIA."
Good links. The NYT article has criticism, as well as information about failed predictions:
While Bueno de Mesquita has published many predictions in academic journals, the vast majority of his forecasts have been done in secret for corporate or government clients, where no independent academics can verify them. “We have no idea if he’s right 9 times out of 10, or 9 times out of a hundred, or 9 times out of a thousand,” Walt says. Walt also isn’t impressed by Stanley Feder’s C.I.A. study showing Bueno de Mesquita’s 90 percent hit rate. “It’s one midlevel C.I.A. bureaucrat saying, ‘This was a useful tool,’ ” Walt says. “It’s not like he’s got Brent Scowcroft saying, ‘Back in the Bush administration, we didn’t make a decision without consulting Bueno de Mesquita.’ ” Other academics point out that rational-actor theory has come under increasing criticism in recent years, as more evidence accumulates that people make many decisions irrationally.
And it’s true that there have been cases when Bueno de Mesquita’s model has gone awry. In his 1996 book, “Red Flag Over Hong Kong,” he predicted that the press in Hong Kong “will become largely a tool of the state” — a highly debatable claim today. (In 2006, Reporters Without Borders noted concerns about self-censorship but said that “journalists remain free in Hong Kong.”) In early 1993, a corporate client asked him to forecast whether the Clinton administration’s health care plan would pass, and he said it would.
Your link to the NYT was hijacked, it now goes to a "anti-viruses" site - I closed the tab as quickly as I could and didn't catch much information about the site. Rather, the NYT site was apparently hijacked - I got redirected.
Works for me, right after you added this comment. The URL is fine. Perhaps you have been hacked or infected with some malware that is responsible for redirecting your browser. The URL is http://www.nytimes.com/2009/08/16/magazine/16Bruce-t.html?pagewanted=all, which is definitely a New York Times URL.
I actually got the virus redirect thing on a different NY Times link earlier today and I'm running Firefox 3 on Ubuntu.
I'm running Firefox 3.5; but the anti-viruses redirect is gone now, I just got through and read the article. I deleted my original comment, no sense in cluttering the thread unnecessarily, so I don't know if this is going to show up or not. anonym's responding comment disappeared from my browser. EDIT: Now it has reappeared along with Kevin's and this new comment (I hate software).
This is a bizarre claim:
He foresaw that China would reclaim Hong Kong 12 years before it happened
It's might be just a typo, but I penalize BdM for it. Britain and China negotiated the hand-over 13 years ahead, so 12 year predictions are unimpressive. Wikipedia dates the negotiation to the governor's 1979 trip to the mainland. (This comment seems to acknowledge but deny that version of history and instead put the focus on 15 year mortgages, which does match the 1982 beginning of serious negotiations. Yet earlier citizenship bills suggest that UK was planning for the handover.)
Do we have any serious hard-to-fake evidence that this is true? Examples given might all well be cherry-picked datapoints.
hard-to-fake evidence
The CIA report, which I haven't read, is pretty hard to fake. But it may not be so useful if we don't know how good is the baseline of CIA analysts and the questions. Tetlock says that it's very easy to write very simple computer programs that beat all of the people he assessed.
I read somewhere that both the CIA analysts and Bueno de Mesquita got 90%, whatever that means, but that he was more precise. I don't know how well-calibrated 90% is, but I suspect that the CIA analysts had more humility than any of Tetlock's experts, perhaps because they considered themselves cogs, rather than celebrities.
I assume he really did publish lots of predictions. But I wouldn't be shocked if he's blatantly lying and they're just wrong. Here's an example of someone contradicted about his predictions by his publications. In 1987, Michael Lewis begins his epilogue to Liar's Poker with "I didn't think the firm was doomed. I didn't think that Wall Street would collapse." Today, he writes "In the two decades since then, I had been waiting for the end of Wall Street." These quotes are not entirely fair, being cherry-picked from the midst of qualifications and explanations. But the similar language is striking.
(When I invoke Tetlock, I'm talking about his book. I don't know what he wrote about BdM.)
Tetlock says that it's very easy to write very simple computer programs that beat all of the people he assessed.
This is probably related to the fallacy of college admissions. Everyone in admissions thinks they can do better a job of predicting college success by using more criteria and their own professional judgement, and in every test using the SAT alone does better in aggregate.
For more on this and related phenomena, see Michael Bishop, "In Praise of Epistemic Irresponsibility: How Lazy And Ignorant Can You Be?"
in every test using the SAT alone does better in aggregate.
This is emphatically wrong. From the most pro-SAT source I can think of, SAT's + High School GPA are the most accurate predictor of first-year grades available. I admit the possibility that if you use a different measure of college success, my point is not valid.
It may be comforting to overstate the value of standardized testing, but that does not make it true, or excuse unfounded generalizations. One should be particularly careful to avoid such generalizations in a blue-green issue like the value of intelligence testing.
I think billswift's point was that the SAT is a better predictor as opposed to using human judgement as in reading the application letter, etc... It seems plausible that adding another "objective" measure like the HSGPA will further increase accuracy. But the big point is that using human judgement will DECREASE accuracy. The really interesting paper is the following as suggested by Alicorn:
Michael Bishop, "In Praise of Epistemic Irresponsibility: How Lazy And Ignorant Can You Be?"
I don't remember my sources, but I came across this in several published books in the 1990s (I think they were from the late 1980s and early 90s but I'm not sure). This was before they weakened the SAT, so maybe that accounts for the difference, or maybe you're going by ad copy, rather than cognitive psych reports, which are attempting to show the value of the tests without offending the consumers of the tests.
In one episode of EconTalk he said he had published a prediction of who Khomeini's successor would be years before the fact.
If the details in Good Magazine are correct, then it looks like costly signaling, not cherry-picking.
The details are that he was condemned at the APSA meeting so harshly that he got a public apology five years later when his prediction turned out to be correct. If he were making tons of controversial predictions and getting publicly condemned every meeting, the Iran expert should have felt no need to apologize when he's right once. It's important for this story that APSA is all of political science, not just Iran.
But these details look hard to check. I'm inclined to throw them out and call it cherry-picking.
I was hoping somebody would be familiar with his work from before. I didn't go hunting around for his papers, since I'm no expert on international politics and would have to spend as much if not more time figuring out what the outcomes were than researching his actual predictions.
Is he doing some of the best work in the area? Perhaps. Is he as good as some say? Definitely not.
As he sometimes acknowledges, the information he feeds into his models is incredibly important. He gets that information by interviewing experts in the field. One might be able to do better than any individual expert by merely averaging the opinions of several experts.
Good hour long interviews with him here: http://www.econtalk.org/archives/_featuring/bruce_bueno_de_mesquita/
You certainly can often do better than individual experts by averaging experts, but BDM's methods actually do add something.
To the extent that BDM predicts things like the Khomeini succession (and something about the Indian parliament, if I recall one of my political economy classes) in ways nobody else sees, including himself, it's worth understanding just how game theory allows unexpected or unintuitive results from completely basic inputs.
BDM's use of experts is confined to the inputs--a place where their judgments are more straightforward. It is rather easier to compare the relative power of two parliamentarians, for instance, than to predict the outcome of a multiparty election. Averaging experts' opinions on the latter may not improve forecasts much, and that's where BDM's innovations have been useful.
I should note too that he sometimes derives the model inputs himself.
I don't want to oversell his abilities, but he's doing pretty innovative work at the cutting edge of political economic theory.
To clarify the last point, I particularly recommend examining the selectorate theory in The Logic of Political Survival, chapters 2 and 3, and "Political Survival and Endogenous Institutional Change" in Comparative Political Studies 2008.
Both are from a recent course I took on political economy.
Here are a couple of things that annoy me. I think he's misleading about a lot of aspects of his methods, yet I don't doubt his his claims about his record, though I think we're missing a lot of context to assess it.
The India example in the NYT implies that he's using the computer to solve the NP problem of finding good coalitions. But once he has the coalition in hand, he should be able to take it to other people--that's the point of NP--and convince them that it's better than the one they suggested, yet this is exactly what he doesn't do in the India example. In fact, he claims not to understand the output himself! This claim about his methods also doesn't seem to match the dynamic point of view, as in his predictions about the path that Iranian politicians will travel before they reach equilibrium. (Part of that is that some of the goals are spread over time, like a certain amount of international attention. But he also implies at other times that he's predicting not just the best coalition, but how people will go about finding it. This is absurd.)
I liked this podcast about the work that Stephen Walt praises in NYT. I particularly liked the examples of Leopold II of Belgium and Chiang Kai-Shek as examples of people who ruled two countries each, with different results; and in Leopold's case, he ruled the two simultaneously. Sometimes BdM implies that this work is related to his predictions, but it doesn't seem so to me from other things he says. In particular, in his academic work, he seems to discount diversity of goals, while when he talks about Iran, he seems to claim that he's taking into account diversity of how much the actors care about the national glory of having a bomb, etc., rather than just caring about its reflection on themselves.
Well, in 21 months we'll be able to have some partial track record:
Those predictions are terribly vague. He produces an illusion of precise predictions about the timing of the power of various factions, as in graphs in his TED talk, but gives no explanation of what these numbers mean. It's not even clear who are the "moneyed interests." It's probably clear who are the different factions of mullahs, but how can we verify that one group has gained power? All this leaves of his prediction is that of the three choices of nothing nuclear, civilian power, and publicly endorsing weapons, he chooses the middle path, the status quo. (there's also the part about weapons grade uranium. I guess that's something.) Maybe always predicting the status quo gets you a better track record than most experts, but it's not a good enough track record to care.
The fact that he does marketing, is not evidence against his prediction abilities, but it is important not to be distracted by the marketing.
I stumbled upon an article called The New Nostradamus, reporting of a game-theoretic model which predicts political outcomes with startling effectiveness. The results are very impressive. However, the site hosting the article is unfamiliar to me, so I'm not certain of the article's verity, but a quick Google seems to support the claims, at least on a superficial skimming. Here's his TED talk. The model seems almost too good to be true, though. Anybody know more?
Some choice bits from the article:
The claim:
The results:
Gets good money for it:
The method should be of special interest to the OB/LW audience, as it brings to mind discussions about self-deception and evolutionary vs. acknowledged goals and behavior: