The Real Lesson of 2012
Philip Tetlock and Dan Gardner.
Liberals had good reason to be delighted with the presidential election of 2012.
Not only did most polls accurately foresee a victory for Barack Obama, the poll-based analysis of Nate Silver and others proved to be spot-on. Conservative pundits were humiliated. All their detailed explanations for why a Romney victory was coming, all their careful dissection of polling methodologies, their talk of data bias, of unquantifiable factors, of gut feelings — it all looked foolish when the votes were counted.
For liberals, it was hard not to gloat. “Let conservatives listen to people with ‘gut feelings,’ and vibration sensibilities,” wrote liberal blogger Markos Moulitsas. “We’ll stick to arithmetic.” Conservatives are ideologically blinkered. Conservatives are anti-science, anti-evidence, anti-reason. Conservatives are so unlike liberals.
Liberals are smart and reality-based and would never fall for the sort of delusions that befuddled conservatives in the weeks leading up to the election.
For liberals, that’s a pleasant conclusion. But it’s wrong. And it’s dangerous.
In a classic study published in 1979, psychologists Charles Lord, Lee Ross, and Mark Lepper carefully selected a group of 48 university students. Half were chosen because they supported capital punishment and believed it was an effective deterrent against crime. The other half felt the opposite.
The researchers asked the students to read a summary of what the researchers said was a randomly selected study. For example, one read: “Kroner and Phillips (1977) compared murder rtates for the year before and after adoption of capital punishment in 14 states. In 11 of the 14 states, murder rates were lower after adoption of the death penalty. This research supports the deterrent effect of the death penalty.”
They were then asked to indicate whether they were now more or less opposed to capital punishment and whether they were more or less convinced that capital punishment deters crime.
The students were then given detailed information about the study, its methodology, criticisms of the methodology, and the author’s responses to the criticisms. And again the students were asked: Are you now more or less opposed to capital punishment? Are you now more or less convinced that capital punishment deters crime?
The whole procedure was then repeated with a second study.
In reality, the studies weren’t randomly selected. One supported the efficacy of capital punishment, one didn’t. And the studies were fakes. The researchers carefully composed them, and arranged their presentation, so they were counter-balanced: same methodologies, same criticisms, same rebuttals. Only the conclusions varied.
And yet all the test subjects became more convinced they were right.
The cause of this seemingly impossible outcome was simple: People were very uncritical and accepting of the study that supported their existing belief but they were very critical and dismissive of the one that did not. Psychologists call this “confirmation bias.” It’s ubiquitous in how people think. And remarkably potent.
Unfortunately, people are very good at hiding the influence of confirmation bias, from others and from themselves. Consider that, as blatant as the students’ biases in the study were, the language they used when commenting on the studies appeared perfectly objective and dispassionate. “The experiment was well thought out, the data collected was valid, and they were able to come up with responses to all criticisms,” a supporter wrote. “The evidence given is relatively meaningless without data about how the overall crime rate went up in those years,” a critic insisted.
In mid-October, following Mitt Romney’s decisive win in the first presidential debate, Barack Obama’s numbers fell and liberals flocked in vast numbers to the websites of Nate Silver and other aggregators. The president is still more likely than not to be re-elected, the quants said. Liberals smiled and their knotted muscles relaxed. It was like a big gulp of bourbon. No one knows how how many gave serious and careful consideration to polling methodologies and statistical analysis before accepting the quants’ conclusions, but it’s probably safe to assume it was a small minority. A very small minority.
Not conservatives, though. They buried themselves in the minutiae of data gathering and collectively developed an impressively elaborate explanation for why the polls were wrong, the quants were fools, and Romney was headed to victory. Post-election, it’s easy to see this as the mental gymnastics of committed partisans desperate to believe. But lots of liberals uncritically embraced the same analyses that conservatives hyper-critically rejected. What is that if not the flip-side of confirmation bias?
And what if the circumstances had been reversed? In the latter stages of the 2004 election, they were, albeit imperfectly. And what we saw then was conservatives uncritically accepting polls while many liberals carefully dissected their methodologies, discovered they were flawed, and came to the happy conclusion that John Kerry’s chances of winning were much better than they appeared.
“Electoral College projections based on state polls show a dead heat. Prjections assuming that undecided voters will break for the challenger in typical proportions give Mr. Kerry more than 300 electoral votes,” wrote New York Times columnist Paul Krugman on October 22, 2004. But the average person wasn’t hearing the truth, Krugman complained, because CNN and Fox were highlighting polls that are methodologically skewed.
And let’s not forget the masterpiece of critical thinking published in 1972, shortly before Richard Nixon’s landslide win over George McGovern. It was a slim book entitled How McGovern Won the Presidency and Why the Polls Were Wrong.
It’s a human thing. And liberals are human, too.