Rethinking the Scientific Method from the New Yorker

The Truth Wears Off

This article from the most recent New Yorker looks at something called “the decline” effect. It is a phenomenon where an established result in science, or “fact” becomes less demonstrable over time. They give examples of antipsychotic medications,certain cognition effects and stents (ahem.) They mention several possible reasons for this effect. Wishful thinking on the part of researchers, for example. They cite acupuncture studies from 1990 to 2000 where all of the results from Asia were positive, when only 50% from the U.S. and Europe were positive. (This was obvious to anyone paying attention to the acupuncture literature at this time. I still ignore most of the studies out of Asia, China in particular.) Another possible explanation was just random chance that lead to the initial results that could not be reproduced.
The problem is, of course, that studies are very expensive! It’s easy enough to call for more studies, but much more difficult to actually procure the money to pay for adequate studies. It was fortunate for the acupuncture community that the Insurance Companies in Germany were willing to pay for large enough studies to establish acupuncture efficacy for knee, hip and back pain. They did not discuss the corruption of studies by the drug companies as referenced in a previous post.

(To read more about acupuncture and my practice, please click here.)

ABSTRACT: ANNALS OF SCIENCE about the decline effect. On September 18, 2007, a few dozen neuroscientists, psychiatrists, and drug-company executives gathered in a hotel conference room in Brussels to hear some startling news. It had to do with a class of drugs known as atypical or second-generation antipsychotics, which came on the market in the early nineties. The therapeutic power of the drugs appeared to be steadily falling. A recent study showed an effect that was less than half of that documented in the first trials, in the early nineties. Before the effectiveness of a drug can be confirmed, it must be tested again and again. The test of replicability, as it’s known, is the foundation of modern research. It’s a safeguard
for the creep of subjectivity. But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts are losing their truth. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. When Jonathan Schooler was a graduate student at the University of Washington, he discovered a surprising phenomenon having to do with language and memory that he called verbal overshadowing. While Schooler was publishing his results in journals, he noticed that it was proving difficult to replicate his earlier findings. Mentions psychologist Joseph Banks Rhine, who conducted several experiments dealing with E.S.P. In 2004, Schooler embarked on an imitation of Rhine’s research in an attempt to test the decline effect. The most likely explanation for the decline is an obvious one: regression to the mean. Yet the effect’s ubiquity seems to violate the laws of statistics. Describes Anders Møller’s discovery of the theory of fluctuating asymmetry in sexual selection. Mentions Leigh Simmons and Theodore Sterling. Biologist Michael Jennions argues that the decline effect is largely a product of publication bias. Biologist Richard Palmer suspects that an equally significant issue is the selective reporting of results—that is, the subtle omissions and unconscious misperceptions, as researchers struggle to make sense of their results. Mentions John Ioannidis. In the late nineteen-nineties, neuroscientist John Crabbe investigated the impact of unknown chance events on the test of replicability. The disturbing implication of his study is that a lot of extraordinary scientific data is nothing but noise. This suggests that the decline effect is actually a decline of illusion. Many scientific theories continue to be considered true even after failing numerous experimental tests. The decline effect is troubling because it reminds us how difficult it is to prove anything.

Read more http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer?printable=true#ixzz17XfGb9b7