Peer reviewers aren’t on the same page a third of time

Study led by Angela Dobele of RMIT finds there is no consensus on the value of a paper in 32 per cent of cases

June 18, 2015
Man and woman holding yes/no cards in front of faces

Academic peer reviewers reach completely different conclusions in nearly one case in every three, research suggests.

A study that examined double-blind feedback on conference papers and journal articles in the UK and Australia found that two reviewers were in exact or approximate agreement in 68 per cent of cases: for example, both recommending rejection, or one advocating acceptance without revision and the other supporting acceptance with minor changes.

This meant that reviewers disagreed in 32 per cent of cases, including some where one assessor recommended acceptance without changes and the other called for rejection.

Author Angela Dobele, deputy head of research and innovation at Melbourne’s RMIT University, writes that this has time implications for track chairs at academic conferences and journal editors, who have to take a final decision.

“If this process is not undertaken, perhaps in the interest of saving time, a manuscript could be rejected instead of engaging with reviewers and resolving irrelevant or conflicting comments,” Dr Dobele says.

The study, which covered 231 submissions to six journals, two conferences and one edited book, also evaluated readers’ comments.

In 15 cases, reviewers who recommended acceptance with major revisions provided less than half a page of comments. Recommendations to reject a submission were accompanied by no feedback in three cases, very little in eight and fewer than six paragraphs in 36 cases.

Although the majority of the feedback provided was constructive, the study classes 15 per cent of comments as being “unconstructive” or neutral, offering little direction for authors seeking to improve their papers. One reviewer wrote simply: “This is crap.”

Writing in Higher Education Research & Development, Dr Dobele suggests that formal training in reviewing could help to improve standards.

She also suggests that the sector might need to consider paying reviewers, particularly if time spent on the process is not counted in academic workloads.

“The nature of the publishing game is only going to get more competitive,” Dr Dobele told Times Higher Education. “Peer review is at the heart of this game, so we need to think about this carefully, thoroughly and urgently.”

chris.havergal@tesglobal.com

You've reached your article limit.

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

POSTSCRIPT:

Article originally published as: Reviewers aren’t on same page a third of time (18 June 2015)

Reader's comments (2)

You allude to the same grades being the best outcome. Which article would you rather read – one that someone loves and someone else hates, or one that a few people think is OK? @dbcallaghan
It’s not unsurprising that reviewers disagree on the merits of a manuscript for publication. Perhaps more interesting is how the discerning Editor handles the disagreement -- do the reviewers cover different topics, has one reviewer missed something? Is one reviewer being overly harsh? -- and thinks through the appropriate next step to take. The fact that 15% of reviewer comments were deemed ‘unconstructive’ certainly lends support for open peer review, where reports are signed and authors (and the reading public) know who the reviewers were. We’ve found that reviewer reports that are open tend to be more constructive than anonymous reports, and substantiate their comments with evidence from the manuscript under review (http://f1000.com/posters/browse/summary/1094564). Certainly training in peer review can only be advantageous, as argued for here (http://www.biomedcentral.com/1741-7015/12/128).

Have your say

Log in or register to post comments

Featured Jobs

Most Commented

men in office with feet on desk. Vintage

Three-quarters of respondents are dissatisfied with the people running their institutions

A group of flamingos and a Marabou stork

A right-wing philosopher in Texas tells John Gill how a minority of students can shut down debates and intimidate lecturers – and why he backs Trump

A face made of numbers looks over a university campus

From personalising tuition to performance management, the use of data is increasingly driving how institutions operate

students use laptops

Researchers say students who use computers score half a grade lower than those who write notes

Canal houses, Amsterdam, Netherlands

All three of England’s for-profit universities owned in Netherlands