Lies, Damned Lies, and Medical Science

Lies, Damned Lies, and Medical Science is the title of a very interesting article by David H. Freedman in “The Atlantic.” It is based on an interview with a Greek doctor, John Ioannidis, and some of his associates.



Have you ever picked up the paper in the morning and read about a study that claimed that consuming more of some vitamin or mineral would make you happier and live longer? Have you ever walked away thinking “Didn’t I one time read that too much of that vitamin or mineral is bad for me?” The answer is often “Yes!“ But you needn’t worry the fact that the two studies seem to be inconsistent. The odds are they are both wrong anyway.


The reason Dr. Ioannidis makes such a good interview for the author is because Ioannidis has devoted his career to outing sloppy, biased, corrupt, and incompetent researchers and their studies. He became interested in the problem as a physician-researcher in the early 1990s while at Harvard. His specialty was to be diagnosing rare diseases. Clearly one would have to search hard for solid information on rare diseases. What he discovered in his initial studies was that not only could he not find good data to support his activities, but he could not find data to support many of the treatment decisions that were commonly made by doctors. This realization motivated him to change course.

“He’s what’s known as a meta-researcher, and he’s become one of the world’s foremost experts on the credibility of medical research. He and his team have shown, again and again, and in many different ways, that much of what biomedical researchers conclude in published studies—conclusions that doctors keep in mind when they prescribe antibiotics or blood-pressure medication, or when they advise us to consume more fiber or less meat, or when they recommend surgery for heart disease or back pain—is misleading, exaggerated, and often flat-out wrong. He charges that as much as 90 percent of the published medical information that doctors rely on is flawed. His work has been widely accepted by the medical community; it has been published in the field’s top journals, where it is heavily cited; and he is a big draw at conferences. Given this exposure, and the fact that his work broadly targets everyone else’s work in medicine, as well as everything that physicians do and all the health advice we get, Ioannidis may be one of the most influential scientists alive. Yet for all his influence, he worries that the field of medical research is so pervasively flawed, and so riddled with conflicts of interest, that it might be chronically resistant to change—or even to publicly admitting that there’s a problem.”

Studies of medical effectiveness are extremely complicated and subject not only to misinterpretation, but also to a bias applied by the researchers. Sometimes this bias is unconscious or inadvertent, but, perhaps, sometimes it isn’t.

“....suggested a bigger, underlying dysfunction, and Ioannidis thought he knew what it was. ‘The studies were biased,’ he says. ‘Sometimes they were overtly biased. Sometimes it was difficult to see the bias, but it was there.’ Researchers headed into their studies wanting certain results—and, lo and behold, they were getting them. We think of the scientific process as being objective, rigorous, and even ruthless in separating out what is true from what we merely wish to be true, but in fact it’s easy to manipulate results, even unintentionally or unconsciously. ‘At every step in the process, there is room to distort results, a way to make a stronger claim or to select what is going to be concluded,’ says Ioannidis. ‘There is an intellectual conflict of interest that pressures researchers to find whatever it is that is most likely to get them funded’.”

Ioannidis wanted to convincingly demonstrate to the medical community that the issues he was raising were not isolated examples that he had cherry-picked to prove a point. He wanted to show the community that they had a problem and they had better recognize it.

“He zoomed in on 49 of the most highly regarded research findings in medicine over the previous 13 years, as judged by the science community’s two standard measures: the papers had appeared in the journals most widely cited in research articles, and the 49 articles themselves were the most widely cited articles in these journals. These were articles that helped lead to the widespread popularity of treatments such as the use of hormone-replacement therapy for menopausal women, vitamin E to reduce the risk of heart disease, coronary stents to ward off heart attacks, and daily low-dose aspirin to control blood pressure and prevent heart attacks and strokes. Ioannidis was putting his contentions to the test not against run-of-the-mill research, or even merely well-accepted research, but against the absolute tip of the research pyramid. Of the 49 articles, 45 claimed to have uncovered effective interventions. Thirty-four of these claims had been retested, and 14 of these, or 41 percent, had been convincingly shown to be wrong or significantly exaggerated. If between a third and a half of the most acclaimed research in medicine was proving untrustworthy, the scope and impact of the problem were undeniable. That article was published in the Journal of the American Medical Association.”

What is worse than realizing that much of what you believed about medicine is incorrect? What is worse is the realization that there is not much you can do to correct the situation.

“’Even when the evidence shows that a particular research idea is wrong, if you have thousands of scientists who have invested their careers in it, they’ll continue to publish papers on it,’ he says. ‘It’s like an epidemic, in the sense that they’re infected with these wrong ideas, and they’re spreading it to other researchers through journals’.”

“But even for medicine’s most influential studies, the evidence sometimes remains surprisingly narrow. Of those 45 super-cited studies that Ioannidis focused on, 11 had never been retested. Perhaps worse, Ioannidis found that even when a research error is outed, it typically persists for years or even decades. He looked at three prominent health studies from the 1980s and 1990s that were each later soundly refuted, and discovered that researchers continued to cite the original results as correct more often than as flawed—in one case for at least 12 years after the results were discredited.”

So what is the lesson learned here? Don’t trust your newspaper? Don’t trust your doctor? Of course you shouldn’t trust them! You should have been there long ago! But now you can mistrust them with much more confidence.