Wisdom Wednesday: How Reliable Are Medical Studies? Half of Findings Couldn’t Be Replicated

Independent researchers couldn’t reproduce the findings of more than half of 100 experiments previously published in three prominent psychology journals, a new review reports.

This review should fuel skepticism over scientific claims, particularly if those claims are based on shaky statistics, said one of the new study’s authors, Brian Nosek, a professor of psychology at the University of Virginia. Nosek is also executive director of the Center of Open Science, the non-profit group that coordinated the project.

Only 47% of the follow-up studies were able to reproduce the same effects of the original studies, the review found. The strength of findings found during original studies also appeared to diminish when successfully replicated, Nosek said.

The new review also calls into question the statistics used in the original studies. About 97% of the original studies showed a statistically significant result, but only 36% of the replication studies did the same.

“Reproducibility is a central feature of science,” Nosek said. “A scientific claim doesn’t become believable because of the status or authority of the person that generated it. Credibility of the claim depends in part on the repeatability of its supporting evidence.”



The study was published in the August 28 issue of Science.

My Take:
My blogs frequency question the validity of a particular study. Most often the study is poorly conceived and poorly structured. This is especially true when the goal is a preconceived outcome. Research sponsored by drug companies are the worst offenders.

I have an upcoming blog on e-cigarettes that illustrates how a poorly structured study skews the results. Please take a few minutes to read it. I think it will really change your thinking.

Don’t accept any study at face value. Check to see who funded the study. This new study was funded by a non-profit group with no ax to grind. I see no conflict there. Study authors generally disclose any associated financial relationship, like doing a study on tobacco while in the employment of the tobacco industry. PubMed will publish those relationships in the abstract, but they are not disclosed in the press releases.

Press releases are the basis for reporting to the general public. When you watch the latest health report on national news, you are getting the watered down version. My news feeds from PubMed, New England Journal of Medicine, the New York Times, and WebMD are much the same. If however, I question any aspect of the report, I review the original study. You would be surprised how certain aspects of the study are highlighted while others are ignored.

The Bottom Line:
Beware of scientific claims. If you really question a study, send it to me. I’ll do the research on the research and if it’s noteworthy, I’ll write a blog about it.

Source: August 27, 2015 National Institutes of Health