Psychology reproducibility ‘crisis’ overstated, new report claims

But authors of original paper hit back, saying latest assessment is ‘very optimistic’

March 4, 2016
Brain scan
Source: iStock

Academics have continued to trade blows over the state of psychology research following the release of a paper questioning the results of a major project that cast doubt on reproducibility in the field.

In August 2015, an attempt to reproduce 100 prominent papers by the Center for Open Science found that only 36 per cent produced statistically significant results, stoking concerns about scientific reliability that have also engulfed biomedicine.

But today saw a group of researchers from Harvard University and the University of Virginia respond with claims that the study contained several statistical errors and failed to repeat the experiments properly.

It lists a number of what it claims are discrepancies between the original studies and attempts at replication.

“An original study that measured Americans’ attitudes toward African-Americans was replicated with Italians, who do not share the same stereotypes; an original study that asked college students to imagine being called on by a professor was replicated with participants who had never been to college,” says the comment, published in Science.

It also criticises the study for only having attempted to reproduce each study once, leading to a much lower rate of replication.

The study “seriously underestimated the reproducibility of psychological science”, it concludes.

But the authors of last year’s replication study have hit back themselves in Science, calling today’s critical paper a “very optimistic assessment” that is “limited by statistical misconceptions and by causal inferences from selectively interpreted, correlational data”.

The corresponding author from last year’s study, Brian Nosek, a psychology professor at Virginia, also took aim at some of the reporting on today’s paper attacking his findings.

One journalist “talks to me at 9.20p saying she only read the press release – not original article, comment or response. Files story @ 9:30p,” he wrote on Twitter.

Other commentators have questioned whether today’s rebuttal means that all is well in psychology.

“The Reproducibility Project is far from the only line of evidence for psychology’s problems. There’s the growing list of failures to replicate textbook phenomena,” wrote Ed Yong in The Atlantic.

“There’s publication bias – the tendency to only publish studies with positive results, while dismissing those with negative ones. There’s evidence of questionable research practices that are widespread and condoned,” he argued.

david.matthews@tesglobal.com

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Featured Jobs

Most Commented

United Nations peace keeper

Understanding the unwritten rules of graduate study is vital if you want to get the most from your PhD supervision, say Kevin O'Gorman and Robert MacIntosh

David Parkins Christmas illustration (22 December 2016)

A Dickensian tale, set in today’s university

Eleanor Shakespeare illustration (5 January 2017)

Fixing problems in the academic job market by reducing the number of PhDs would homogenise the sector, argues Tom Cutterham

Houses of Parliament, Westminster, government

There really is no need for the Higher Education and Research Bill, says Anne Sheppard

poi, circus

Kate Riegle van West had to battle to bring her circus life and her academic life together