More than half of psychology papers are not reproducible

Initiative to replicate findings of 100 prominent studies raises further questions about health of discipline

August 27, 2015
Duplicates of man's face

A landmark initiative to reproduce the findings of 100 prominent psychology papers has been successful in less than half of cases.

The Reproducibility Project: Psychology was launched in 2011 by the Center for Open Science, a US non-profit organisation, in the wake of a number of fraud scandals in psychology, such as that involving Diederik Stapel, who admitted in 2011 to faking more than 50 papers.

One hundred papers were chosen from 2008 issues of three important journals in psychology, and about 350 scientists were involved in meticulous attempts to reproduce them.

Replication attempts used powerful statistical methods, and approaches were independently reviewed and posted on a central repository. The authors of the original studies were also consulted regularly.

The results of the project are reported in a paper, “Estimating the reproducibility of psychological science”, published in the journal Science.

Based on five different measures of reproducibility, the paper concludes that reproduction was successful in less than half of cases. For instance, while 97 per cent of the original studies reported statistically significant results, only 36 per cent of replications did so.

Replication rates were particularly low in social psychology, compared with cognitive psychology.

The paper also reports that even where the original effects were replicated, the statistical size of the reproduced effect was typically less than half that of the original.


Reproducing results: how big is the problem?


Brian Nosek, professor of psychology at the University of Virginia and executive director of the Center for Open Science, cautioned against concluding either that reproduced studies were thereby proved, or that those that could not be reproduced must have reported “false positives”. It was also possible, he said, that the replication could be a false negative, or that the original study and its replication had important methodological differences.

“One reason for the latter would be if the replication team did not implement the procedure correctly or just did a terrible job conducting the study,” he said – although he noted that every effort had been made to maximise rigour.

Gilbert Chin, senior editor at Science, described the findings as “somewhat disappointing”, but emphasised that “the outcome does not speak directly to the validity or the falsity of the theories” underlying the experiments.

“What it does say is that we should be less confident about many of the original experimental results that were provided as empirical evidence in support of those theories,” he said.

Alan Kraut, executive director of the Association for Psychological Science and a board member of the Center for Open Science, noted that even statistically significant “real findings” would “not be expected to replicate over and over again…The only finding that will replicate 100 per cent of the time is likely to be trite, boring, and probably already known.”

Professor Nosek conceded that there was “a lot of room to improve reproducibility”, but he did not see the paper as telling a “pessimistic” story.

“The project is a demonstration of science demonstrating an essential quality: self-correction,” he said. “A community of researchers volunteered their time to contribute to a large project for which they would receive little individual credit…It shows that many scientists embrace scientific ideals and demonstrate their commitment to those by reflecting them in daily practice.”

Moreover, he cautioned, the project was “a drop in the bucket in terms of trying to get a precise estimate of the overall reproducibility of a discipline, let alone of science more generally”.

paul.jump@tesglobal.com

You've reached your article limit.

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

POSTSCRIPT:

Print headline: Majority of psychology papers are not reproducible, study discovers

Reader's comments (1)

If psychology is such an inexact social science, is it a science at all? - or if “the outcome does not speak directly to the validity or the falsity of the theories”, what was the point of the exercise?

Have your say

Log in or register to post comments

Featured Jobs

Most Commented

Monster behind man at desk

Despite all that’s been done to improve doctoral study, horror stories keep coming. Here three students relate PhD nightmares while two academics advise on how to ensure a successful supervision

opinion illustration

Eliminating cheating services, even if it were possible, would do nothing to address students’ and universities’ lack of interest in learning, says Stuart Macdonald

Female professor

New data show proportion of professors who are women has declined at some institutions

Sir Christopher Snowden, former Universities UK president, attacks ratings in wake of Southampton’s bronze award

Reflection of man in cracked mirror

To defend the values of reason from political attack we need to be more discriminating about the claims made in its name, says John Hendry