Journal tries ‘results-free’ peer review

BioMed Central psychology journal will attempt to tackle publication bias by withholding research results from peer reviewers

September 23, 2016
Oxford undergraduate taking taste test
Source: Rex
Taste test: the trial will encourage reviewers to judge the research on its premise and methods rather than its outcomes

A psychology journal is launching a trial during which some reviewers will not see the results, discussion and conclusion sections of manuscripts before deciding whether or not to accept them for publication.

The journal hopes that this will encourage reviewers to judge the research on its premise and methods rather than its outcomes. 

Although some researchers see the idea as a new and “important” one, it has raised questions over whether the method could work outside the social sciences.

In academic publishing, it is well known that research results that do not support a hypothesis are less likely to be published than those that do, despite the belief that such “null” results are still important to the scientific endeavour. 

The trial by BMC Psychology, hopes to change this. First, a pilot phase will see the first 10 authors to opt into the scheme go through a results-free peer review process to show that it is viable. Then a randomised controlled trial will begin whereby papers are randomly assigned to either a regular or results-free peer review process.

Liz Bal, associate publisher at BioMed Central, which publishes the journal, said that papers following the results-free process would “be ‘accepted in principle’ based on the rationale and methods only. These manuscripts will then be reviewed again by the same reviewers but with the omitted sections visible.”

Following this, she said that the decision to publish will be changed only if the revealed sections “deviate unjustifiably from the stated aims and methods. We believe that this could help reduce publication bias by basing the decision to publish purely on the scientific rigour of the study design.”

Philip Moriarty, professor of physics at the University of Nottingham, said that he could see the merits of results-free peer review in the social sciences.

“It’s an important idea…but I struggle to see how it’d work in physics or, more broadly, the physical sciences. For us, the experimental techniques can be fairly standard. Therefore, the results – and, of course, their interpretation – are absolutely core to the paper,” he added.

“Anything that improves the quality of published research is of course to be welcomed,” he said. But he feared that it could “lead to some exceptionally boring and practically useless papers if all they do is rule out a particular hypothesis without providing any insight into alternative explanations”.

Apostolos Koutropoulos, associate lecturer of instructional design at the University of Massachusetts Boston’s College of Advancing and Professional Studies, said that the two-stage process seemed a “novel” way of addressing one potential area of peer reviewer bias.

However, he added that it did nothing to address the problems of matching an appropriate reviewer with each paper.

“I think it’s a false assumption to assume that someone who has disciplinary expertise is also a methodological expert [who] can understand [and] adequately critique the article, and offer helpful suggestions to the authors,” he said.

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments

Have your say

Log in or register to post comments

Featured Jobs

Researcher in Fluid Dynamics

Norwegian University Of Science & Technology -ntnu


Greenwich School Of Management Ltd

PhD Research Fellow in Medical Physics

Norwegian University Of Science & Technology -ntnu

Postdoctoral position in Atmospheric and Space Physics

Norwegian University Of Science & Technology -ntnu

PhD Fellow in Machine Learning

Norwegian University Of Science & Technology -ntnu
See all jobs

Most Commented

Doctoral study can seem like a 24-7 endeavour, but don't ignore these other opportunities, advise Robert MacIntosh and Kevin O'Gorman

Matthew Brazier illustration (9 February 2017)

How do you defeat Nazis and liars? Focus on the people in earshot, says eminent Holocaust scholar Deborah Lipstadt

Improvement, performance, rankings, success

Phil Baty sets out why the World University Rankings are here to stay – and why that's a good thing

Warwick vice-chancellor Stuart Croft on why his university reluctantly joined the ‘flawed’ teaching excellence framework

people dressed in game of thrones costume

Old Germanic languages are back in vogue, but what value are they to a modern-day graduate? Alice Durrans writes