Peer reviewers aren’t on the same page a third of time

Study led by Angela Dobele of RMIT finds there is no consensus on the value of a paper in 32 per cent of cases

June 18, 2015
Man and woman holding yes/no cards in front of faces

Academic peer reviewers reach completely different conclusions in nearly one case in every three, research suggests.

A study that examined double-blind feedback on conference papers and journal articles in the UK and Australia found that two reviewers were in exact or approximate agreement in 68 per cent of cases: for example, both recommending rejection, or one advocating acceptance without revision and the other supporting acceptance with minor changes.

This meant that reviewers disagreed in 32 per cent of cases, including some where one assessor recommended acceptance without changes and the other called for rejection.

Author Angela Dobele, deputy head of research and innovation at Melbourne’s RMIT University, writes that this has time implications for track chairs at academic conferences and journal editors, who have to take a final decision.

“If this process is not undertaken, perhaps in the interest of saving time, a manuscript could be rejected instead of engaging with reviewers and resolving irrelevant or conflicting comments,” Dr Dobele says.

The study, which covered 231 submissions to six journals, two conferences and one edited book, also evaluated readers’ comments.

In 15 cases, reviewers who recommended acceptance with major revisions provided less than half a page of comments. Recommendations to reject a submission were accompanied by no feedback in three cases, very little in eight and fewer than six paragraphs in 36 cases.

Although the majority of the feedback provided was constructive, the study classes 15 per cent of comments as being “unconstructive” or neutral, offering little direction for authors seeking to improve their papers. One reviewer wrote simply: “This is crap.”

Writing in Higher Education Research & Development, Dr Dobele suggests that formal training in reviewing could help to improve standards.

She also suggests that the sector might need to consider paying reviewers, particularly if time spent on the process is not counted in academic workloads.

“The nature of the publishing game is only going to get more competitive,” Dr Dobele told Times Higher Education. “Peer review is at the heart of this game, so we need to think about this carefully, thoroughly and urgently.”

chris.havergal@tesglobal.com

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

POSTSCRIPT:

Article originally published as: Reviewers aren’t on same page a third of time (18 June 2015)

Reader's comments (2)

You allude to the same grades being the best outcome. Which article would you rather read – one that someone loves and someone else hates, or one that a few people think is OK? @dbcallaghan
It’s not unsurprising that reviewers disagree on the merits of a manuscript for publication. Perhaps more interesting is how the discerning Editor handles the disagreement -- do the reviewers cover different topics, has one reviewer missed something? Is one reviewer being overly harsh? -- and thinks through the appropriate next step to take. The fact that 15% of reviewer comments were deemed ‘unconstructive’ certainly lends support for open peer review, where reports are signed and authors (and the reading public) know who the reviewers were. We’ve found that reviewer reports that are open tend to be more constructive than anonymous reports, and substantiate their comments with evidence from the manuscript under review (http://f1000.com/posters/browse/summary/1094564). Certainly training in peer review can only be advantageous, as argued for here (http://www.biomedcentral.com/1741-7015/12/128).

Have your say

Log in or register to post comments

Most Commented

question marks PhD study

Selecting the right doctorate is crucial for success. Robert MacIntosh and Kevin O'Gorman share top 10 tips on how to pick a PhD

Pencil lying on open diary

Requesting a log of daily activity means that trust between the institution and the scholar has broken down, says Toby Miller

India, UK, flag

Sir Keith Burnett reflects on what he learned about international students while in India with the UK prime minister

Application for graduate job
Universities producing the most employable graduates have been ranked by companies around the world in the Global University Employability Ranking 2016
Construction workers erecting barriers

Directly linking non-EU recruitment to award levels in teaching assessment has also been under consideration, sources suggest