Casting doubt on the methodology of many well-regarded consumer studies, researchers have found over half of published scientific studies could not be replicated to create the same results as found in the original reports.
A team of 270 scientists tried reproducing 100 studies focused on cognitive and social psychology that had been published in three top U.S. journals in 2008. Only 39 percent came out with the same results as the initial reports, according to findings in the journal Science. The low rates occurred despite the researchers working in close collaboration with the original authors.
Few of the redone studies contradicted the original ones, but their results were not as strong as originally claimed.
The project is part of an effort from the Center for Open Science, a nonprofit technology company directed by Brian Nosek, a psychology professor at the University of Virginia, that aims to increase transparency and reproducibility in scientific research.
Beyond small sample size, a main critique of consumer studies from companies or consultants are that they can be slanted due to skewed survey questions or readings biased to lead to a favored result. The psychology researchers, whether intentionally or unintentionally, were likewise found to be motivated to tweak their results, although for somewhat different reasons.
Seeking tenure, grants or professional acclaim, researchers are under pressure to deliver results that regularly appear in reputable and high-profile journals.
"Not everything we do gets published," Prof. Nosek told the Daily News. "Novel, positive and tidy results are more likely to survive peer review and this can lead to publication biases that leave out negative results and studies that do not fit the story that we have."
To get a more significant result, researchers sometimes tweak experiements or pick only the most favorable data for analysis. By random chance, some experiments may produce results that appear significant. In the reproduction project, geographical factors, such as interviewing people in one country versus another, as well as the time the study took place was said to account for some of the weaker results.
The study called attention to how much bias drives publication in psychology. It called for the increasing need for more replication, although double-checking results doesn't earn much coverage or accolades. Also urged was more extensive explanations of research methods, adequate sample sizes, and wider reporting of studies that show null results or that didn't support the hypothesis.
- Estimating the reproducibility of psychological science - Science
- New study says studies are wrong - Daily News
- Science's Reproducibility Problem: 100 Psych Studies Were Tested and Only Half Held Up - Newsweek
- Over half of psychology studies fail reproducibility test - Nature
- Many Psychology Findings Not as Strong as Claimed, Study Says - The New York Times
- No, science's reproducibility problem is not limited to psychology - The Washington Post