Dan Gardner is the New York Times best-selling author of Risk, Future Babble, Superforecasting (co-authored with Philip E. Tetlock), and How Big Things Get Done (co-authored with Bent Flyvbjerg). His books have been published in 26 countries and 20 languages. Prior to becoming an author, Gardner was an award-winning investigative journalist. More >

Patients' anecdotes are not evidence

Judging by the controversy surrounding the refusal of the federal government to fund clinical trials of the "liberation treatment" of multiple sclerosis, the media and the public have forgotten one of the biggest health stories of the early 1990s. A reminder is in order. The issue then was silicone breast implants. In the 1980s, case studies of women who got sick after getting implants started to pop up in medical journals. Their illnesses were serious -- mostly connective tissue diseases such as rheumatoid arthritis and lupus. News of a lawsuit against manufacturers was widely reported. More stories surfaced in medical journals and the news media. In 1990, CBS journalist Connie Chung produced an hour-long special on the controversy. It was a parade of tragic stories, with one tearful woman after another describing how she got sick after getting the implants. Chung's tone throughout was angry and accusatory. It was obvious that silicone breast implants were killing women. A flood of stories followed. "Toxic Breasts" and "Ticking Time Bombs" were typical headlines. Advocacy groups demanded a ban. Lawsuits started to go against manufacturers. In one, Dow Corning was ordered to pay a woman who claimed her implants sickened her $7.34 million. Under intense pressure, the United States Food and Drug Administration told manufacturers they had 90 days to provide evidence the implants were safe. That was impossible so, in April 1992, the FDA banned them -- while emphasizing that the implants had not actually been proven unsafe and the ban would be lifted if the manufacturers could prove them safe. Activists wanted more. "We are the evidence!" sick women would shout at rallies and in the audiences of talk shows. They were healthy; they got implants; they got sick. And there were thousands of stories like theirs. What more evidence was needed? Journalists were convinced. So was the public. Surveys showed that people considered silicone breast implants to be far more dangerous than a long list of substances and activities. They even deemed having implants to be almost as risky as smoking a pack of cigarettes a day. But the FDA was right. Implants hadn't been proven dangerous. "There are thousands upon thousands of women who have breast implants and complain of terrible pain," an ABC reporter said in 1995. "Can they all be wrong?" Indeed they could. At the time there were roughly 100 million women in the United States. Of those, one per cent had implants. And one per cent had connective tissue diseases. Thus, 10,000 women could be expected to have both implants and disease by coincidence alone. This is a fundamental problem in medicine. Indeed, in all science. People find stories about other people deeply compelling. It's human nature. A story about one suffering woman feels like significant evidence. A collection of such stories feels conclusive. But these stories are quite misleading. Indeed, the actual scientific value of a single anecdote is effectively nil. Even a collection of such stories can prove nothing. "Anecdotes aren't data," scientists like to say. But human nature shrugs. "Data aren't anecdotes," it says. In the early 1990s, journalists and the public went with human nature. Then science started to deliver results: Starting in 1994, one major study concluded silicone breast implants do not cause connective tissue disease. In 2006, after more than a decade of needless fear and suffering, the FDA lifted the ban. Paolo Zamboni's "liberation treatment" is a powerful story. One patient after another comes to the physician who sought a cure for his beloved wife, and one patient after another seems to have the weight of terrible suffering lifted. Nothing could be more moving. Or more intuitively persuasive. But this is not science. And when scientists examined the treatment, they were not impressed. A joint report of the Canadian Institutes of Health Research and the MS Society of Canada issued two weeks ago came to four major conclusions: Zamboni's work is flawed; he has weak evidence; the treatment is dangerous; and there is little reason to think that the theory underlying the treatment is correct. Many commentators accepted all that but insisted that it is unreasonable to deny funding for research on the grounds that there isn't enough evidence. Want more evidence? Fund the research! This would be a compelling argument if "liberation treatment" were the only one of its kind. But it's not. Far from it. Nothing is more common than putative treatments and cures unsupported by solid scientific evidence of efficacy. All that sets Zamboni's treatment apart is the enormous media attention it has garnered. Given that we have only a limited amount of money available for medical research, and given that money spent on clinical trials of one possible treatment is not available for another, we have to direct money where it's most likely to produce positive results. And that doesn't include "liberation treatment." The scientists were right. And the government was right to listen to the scientists. That's deeply dissatisfying, of course. But it is rational. And ultimately, it is rationality, not anecdotes and intuition, that saves lives.