This is the ideal situation. However, what happens when the physical systems trying to be studied are too complex to be able to control variables? When theories and hypothesis abound and there is no definitive way to test them the result is often much less than beautiful. Two such complex systems of great importance are the planet earth and the human body. When complexity combines with critical importance and the potential for power, prestige, and economic gain, one discovers that science is less than rigorous, and scientists suffer from the same human failings as many other professionals.
A recent issue of The Economist examined the reliability of science by devoting two articles to the problem: How science goes wrong, and Trouble at the lab. The articles were meant to refer to science in a general sense, but, tellingly, the examples of interest were all from the medical and social sciences.
The main concern of the articles lies in the growing awareness that research findings that have been published in peer-reviewed journals, and accepted as "scientific truth," cannot be reproduced by other investigators.
Such evidence has led to this conclusion:
The articles indicate a number of reasons why shoddy scientific research finds its way into the published literature.
Considerable discussion is given to the misuse of statistics by researchers. Statistical analysis is rather simple when dealing with something like a flipped coin. There are only two possible outcomes, the initial conditions are well defined, and the final condition (heads or tails) is easily determined. Now consider a test of a medical intervention on a human being. All humans respond differently to a drug; consequently, the results of a study can depend critically on the characteristics of those chosen for testing. Also, there is rarely any result as unequivocal as the heads or tails of a coin toss. It is often difficult to precisely determine the effect of a medication on an individual patient.
A professor Ioannidis addressed the statistical issues inherent in medical research and came to this conclusion:
Ioannidis is a respected expert in evaluating the methods and results of medical science. His work was covered more completely in an article in The Atlantic by David H. Freedman: Lies, Damned Lies, and Medical Science. Ioannidis believes that there is much more wrong with medical science than poor statistical analysis.
It is too easy for research results to contain inadvertent or purposeful biases.
Ben Goldacre lists at least ten ways in which drug trials can be manipulated by those conducting the trial to produce a misleading result. As an extreme suggestion of bias he includes this comment indicating that trial results can be strongly dependent on the financial interest of the researchers in the results.
Medical research is not the only area in which scientific research has had trouble delivering credible results.
Carl Hart has also pointed out that early research into the effects of narcotics and other drugs were often biased in such a way as to produce the results that government funding agencies wished for. This led to major misunderstandings of the nature of human response to such drugs. The results of such biased research included unnecessary criminalization of drug use and the wasteful and ineffective mass incarceration of recreational drug users.
One sees disturbing over-reach in the area of climate modeling. Green house gases are increasing and altering the heat load of the planet. The global temperature is rising. That much is clear. What is uncertain is the prediction of precisely how the planet will respond to this increase in atmospheric carbon load. It is impossible, as of now, to develop a model that can accurately treat all the important physical phenomena. All the important planetary responses have probably not even been identified yet, let alone rendered into accurate models.
The danger is that ambitious scientists, eager to make headlines, are promoting model results and making predictions that are little more than educated guesses. When these predictions fail, the climate naysayers are provided with ammunition to support their dangerous views. This is too serious an issue to allow academic competition to undermine the programs that need to be initiated.
The general public does not read the scientific literature. What it knows of science is generally obtained through the popular media. Newspapers will pick up an article on a scientific result if it is newsworthy or has some entertainment value. Exciting new discoveries are what are being sought. Subsequent studies demonstrating that an exciting new discovery was actually false are not newsworthy and rarely see the light of day. Let the reader beware.
Ben Goldacre is the author of Bad Pharma: How Drug Companies Mislead Doctors and Harm Patients.
Carl Hart is the author of High Price: A Neuroscientist’s Journey of Self-Discovery That Challenges Everything You Know about Drugs and Society.