Modern Scientists Are Wrong Far More Than You Think
Statisticians have shown that many scientific findings are wrong, and without an increase in statistical know-how for scientists it’ll continue happening.
Imagine a hypothetical early scientist who wondered whether or not her tribe’s ritual rain dance had anything to do with whether or not it rained. Our scientist decides on the following simple and elegant experimental design: every day for a year, she wakes up and flips a coin. If it lands on heads, she performs a rain dance; if it lands on tails, she doesn’t. At the end of the day, she records whether or not it rained.
Though a rain dance doesn’t influence the weather, our scientist could, nevertheless, be incredibly unlucky. Perhaps, just by sheer chance, she finds that over the course of the entire year, it happened to rain 99 percent of the times she performed her rain dance. She might reasonably come to the false conclusion that her rain dance causes it to rain.
This is all to say: science is fallible. Even at its best, science can make mistakes. A good scientist will design her experiment so that the chances of arriving at a false conclusion are low, but she can never design a perfect experiment; she always has to live with some small, lingering chance that what looked like compelling data was actually just plain happenstance.
Most reasonable people will begrudgingly accept that any given scientific finding has a small chance of being false. It’s like finding out that the FDA allows 1mg of mouse feces in a pound of black pepper; mouse poop is unsavory, but at least it only makes up 1 part in 450,000. However, there has been a growing concern for a bit over a decade that the state of science is far worse than this. Perhaps most famously, Stanford professor John Ioannidis proclaimed in 2005: “It can be proven that most claimed research findings are false.” Though the disease is severe, the root cause is unassuming. Some statisticians, like Ioannidis, suspect that the primary culprit is simply a widespread confusion about statistics.