A talk with Philip Tetlock
Expertise is overrated
THE DEMOCRATS HAVE nominated a presidential candidate, Senator Barack Obama, with a short resume when it comes to foreign policy. In a countermove, the Republican presidential candidate, Senator John McCain, selected a running mate, Governor Sarah Palin, with less experience than Obama. In this "change" election year, Washington experience is looking like a minus.
Some may feel trepidation at the possibility that a neophyte could soon occupy the White House, but Philip Tetlock, a 54-year-old University of California political psychologist, suggests that experience is overrated. Tetlock has spent two decades asking foreign policy experts to make predictions about world events, and then tracking their accuracy. In that time, he has assembled a database of more than 80,000 individual predictions by 284 experts.
The result: Expertise and experience made very little difference. Experts on the whole barely outperform a coin toss in predicting the future. That's something to keep in mind the next time experts are asked for advice on whether to make war or peace.
The best experts, a group Tetlock calls foxes, can get their success rate up to nearly 60 percent - better than "heads or tails," but not by much. In his 2005 book, "Expert Political Judgment," Tetlock contrasts these foxes, who approach problems from multiple points of view, with another group he calls hedgehogs. The hedgehogs are more ideological, draw conclusions from a single overarching theory, and make more mistakes. The labels stem from an aphorism employed by Isaiah Berlin: "The fox knows many things, but the hedgehog knows one big thing."
Of course, there is more to being an expert than predicting the future. Wisdom can also mean knowing whether to change course, or how to avoid overconfidence in one's ability to understand the situation. These are things that may come with experience.
Tetlock spoke with Ideas by phone from his home in Berkeley.
IDEAS: In your book, you found that an expert's years of professional experience, ranging from one to 36, made no significant difference. How is that possible?
TETLOCK: There are some types of tasks where you wouldn't expect experience to help. If you were asking people to make predictions about whether the little ball on the roulette wheel is going to land in red or black or green, you wouldn't expect that people who have been playing roulette for 50 years would do any better than people who have been playing roulette for 15 minutes.
IDEAS: Is the political prediction game really like roulette?
TETLOCK: My data are consistent with the hypothesis there's a very large [random] component.
IDEAS: So is your message to potential experts: Don't study, don't hit the books, don't visit foreign countries, just wing it, because experience doesn't matter?
TETLOCK: No. I think that would be rash. . . . I think it would be fairer to say that if you have impressive academic credentials and you have an impressive amount of experience, you should be alert to the possibility you could still be profoundly wrong.
IDEAS: I've heard experts defend their profession by saying, "We may not be able to predict the future, but we can explain the past."
TETLOCK: When experts are explaining the past, they actually are very deterministic. They have a considerable facility at generating lots of reasons why things had to work out.
IDEAS: You can always come up with a narrative that explains why your theories are right.
TETLOCK: Indeed. And those narratives often allow people to generate a lot of confidence - false confidence - in their ability to predict the future.
IDEAS: Any individual expert is likely to be wrong. What happens when you put a bunch of them in the same room? Does the consensus come up with a better prediction than the individual?
TETLOCK: Usually yes. Jim Surowiecki wrote that very clever book on "The Wisdom of Crowds." . . . Sometimes social scientists call it the miracle of aggregation. It's an interesting fact that the average of the experts' predictions often outperforms the individuals from whom the average was derived.
IDEAS: But the consensus can be wrong, too, as in the case of the widely shared view that Iraq had weapons of mass destruction. How does that happen?
TETLOCK: The crowd can certainly be wrong. Sometimes, the crowd can be dramatically wrong. . . . One big danger is if you're dealing with an ideologically, intellectually homogeneous group, and they're all talking to each other. They can really talk each other into some quite extreme positions.
IDEAS: Can you give me an example of a question you asked your experts?
TETLOCK: There certainly was an exercise with respect to Iraq, prior to the invasion of Iraq, about how messy it would be and what the outcome would look like one year out, two years out, five years out, 10 years out.
IDEAS: Do you ever try answering the questions you've asked experts yourself, and how have you done?
TETLOCK: I would say it's been consistently humbling. I've made so many errors. I've been off on so many things. I don't even know where to begin.
Rick Heller is a freelance journalist in Belmont. He can be reached at rick.heller@yahoo.com. 