Distrusting Science

Science can be a stimulating, and sometimes beautiful, field of endeavor. One constructs theories or hypotheses about how physical systems work, then the theorist or others test these concepts by performing experiments to determine the validity. When well-posed experiments can be performed, where extraneous variables can be controlled and only those of interest are operative, knowledge is gained and the theories or hypotheses are either supported or not. The physical systems being tested are reproducible so that others can verify the validity of a single experiment. Progress is made, new products may be developed, and more advanced theories ensue.

This is the ideal situation. However, what happens when the physical systems trying to be studied are too complex to be able to control variables? When theories and hypothesis abound and there is no definitive way to test them the result is often much less than beautiful. Two such complex systems of great importance are the planet earth and the human body. When complexity combines with critical importance and the potential for power, prestige, and economic gain, one discovers that science is less than rigorous, and scientists suffer from the same human failings as many other professionals.

A recent issue of The Economist examined the reliability of science by devoting two articles to the problem: How science goes wrong, and Trouble at the lab. The articles were meant to refer to science in a general sense, but, tellingly, the examples of interest were all from the medical and social sciences.

The main concern of the articles lies in the growing awareness that research findings that have been published in peer-reviewed journals, and accepted as "scientific truth," cannot be reproduced by other investigators.

"A few years ago scientists at Amgen, an American drug company, tried to replicate 53 studies that they considered landmarks in the basic science of cancer, often co-operating closely with the original researchers to ensure that their experimental technique matched the one used first time round. According to a piece they wrote last year in Nature, a leading scientific journal, they were able to reproduce the original results in just six. Months earlier Florian Prinz and his colleagues at Bayer HealthCare, a German pharmaceutical giant, reported in Nature Reviews Drug Discovery, a sister journal, that they had successfully reproduced the published results in just a quarter of 67 seminal studies."

Such evidence has led to this conclusion:

"When an official at America’s National Institutes of Health (NIH) reckons, despairingly, that researchers would find it hard to reproduce at least three-quarters of all published biomedical findings, the public part of the process seems to have failed."

The articles indicate a number of reasons why shoddy scientific research finds its way into the published literature.

"Various factors contribute to the problem. Statistical mistakes are widespread. The peer reviewers who evaluate papers before journals commit to publishing them are much worse at spotting mistakes than they or others appreciate. Professional pressure, competition and ambition push scientists to publish more quickly than would be wise. A career structure which lays great stress on publishing copious papers exacerbates all these problems. 'There is no cost to getting things wrong,' says Brian Nosek, a psychologist at the University of Virginia who has taken an interest in his discipline’s persistent errors. ‘The cost is not getting them published’."

Considerable discussion is given to the misuse of statistics by researchers. Statistical analysis is rather simple when dealing with something like a flipped coin. There are only two possible outcomes, the initial conditions are well defined, and the final condition (heads or tails) is easily determined. Now consider a test of a medical intervention on a human being. All humans respond differently to a drug; consequently, the results of a study can depend critically on the characteristics of those chosen for testing. Also, there is rarely any result as unequivocal as the heads or tails of a coin toss. It is often difficult to precisely determine the effect of a medication on an individual patient.

A professor Ioannidis addressed the statistical issues inherent in medical research and came to this conclusion:

"In 2005 John Ioannidis, an epidemiologist from Stanford University, caused a stir with a paper showing why, as a matter of statistical logic, the idea that only one such paper in 20 gives a false-positive result was hugely optimistic. Instead, he argued, "most published research findings are probably false." As he told the quadrennial International Congress on Peer Review and Biomedical Publication, held this September in Chicago, the problem has not gone away."

Ioannidis is a respected expert in evaluating the methods and results of medical science. His work was covered more completely in an article in The Atlantic by David H. Freedman: Lies, Damned Lies, and Medical Science. Ioannidis believes that there is much more wrong with medical science than poor statistical analysis.


"….he worries that the field of medical research is so pervasively flawed, and so riddled with conflicts of interest, that it might be chronically resistant to change—or even to publicly admitting that there’s a problem."

It is too easy for research results to contain inadvertent or purposeful biases.

"‘The studies were biased,’ he says. ‘Sometimes they were overtly biased. Sometimes it was difficult to see the bias, but it was there.’ Researchers headed into their studies wanting certain results—and, lo and behold, they were getting them. We think of the scientific process as being objective, rigorous, and even ruthless in separating out what is true from what we merely wish to be true, but in fact it’s easy to manipulate results, even unintentionally or unconsciously. ‘At every step in the process, there is room to distort results, a way to make a stronger claim or to select what is going to be concluded,’ says Ioannidis. ‘There is an intellectual conflict of interest that pressures researchers to find whatever it is that is most likely to get them funded’."

Ben Goldacre lists at least ten ways in which drug trials can be manipulated by those conducting the trial to produce a misleading result. As an extreme suggestion of bias he includes this comment indicating that trial results can be strongly dependent on the financial interest of the researchers in the results.

"In 2007, researchers looked at every published trial that set out to explore the benefit of a statin. These are cholesterol-lowering drugs which reduce your risk of having a heart attack….This study found 192 trials in total, either comparing one statin against another, or comparing a statin against a different kind of treatment….they found that [drug] industry-funded trials were twenty times more likely to give results favoring the test drug….that’s a very big difference."

Medical research is not the only area in which scientific research has had trouble delivering credible results.

Carl Hart has also pointed out that early research into the effects of narcotics and other drugs were often biased in such a way as to produce the results that government funding agencies wished for. This led to major misunderstandings of the nature of human response to such drugs. The results of such biased research included unnecessary criminalization of drug use and the wasteful and ineffective mass incarceration of recreational drug users.

One sees disturbing over-reach in the area of climate modeling. Green house gases are increasing and altering the heat load of the planet. The global temperature is rising. That much is clear. What is uncertain is the prediction of precisely how the planet will respond to this increase in atmospheric carbon load. It is impossible, as of now, to develop a model that can accurately treat all the important physical phenomena. All the important planetary responses have probably not even been identified yet, let alone rendered into accurate models.

The danger is that ambitious scientists, eager to make headlines, are promoting model results and making predictions that are little more than educated guesses. When these predictions fail, the climate naysayers are provided with ammunition to support their dangerous views. This is too serious an issue to allow academic competition to undermine the programs that need to be initiated.

The general public does not read the scientific literature. What it knows of science is generally obtained through the popular media. Newspapers will pick up an article on a scientific result if it is newsworthy or has some entertainment value. Exciting new discoveries are what are being sought. Subsequent studies demonstrating that an exciting new discovery was actually false are not newsworthy and rarely see the light of day. Let the reader beware.

Ben Goldacre is the author of Bad Pharma: How Drug Companies Mislead Doctors and Harm Patients.

Carl Hart is the author of High Price: A Neuroscientist’s Journey of Self-Discovery That Challenges Everything You Know about Drugs and Society.