A very random peer review

September 15, 2000

The peer review, one of the pillars of modern scientific method, is a perilously random process, according to a new study.

Researchers have carried out detailed analysis of how the opinions of referees used by two leading clinical neuroscience journals tallied with one another, and how this influenced whether a scientific paper was accepted for publication. They also looked at the submission process for two conferences.

Peter Rothwell, a neurologist at the Radcliffe Infirmary, Oxford, and Christopher Martyn, at the Medical Research Council's environmental epidemiology unit, Southampton, found that the incidence of agreement between the expert reviewers as to whether a paper should be accepted for publication, rejected or revised, was not statistically significant.

Rothwell said: "If you gave two chimpanzees pens and got them to stick them into one paper or the other, they'd agree about half the time as well."

The research, published in the latest issue of the journal Brain, backs up work that found similar problems with the peer-review process in fields such as psychology and physical sciences.

The experts compared comments from pairs of independent referees on 179 submitted papers to one journal and 116 to another. They also looked at the conference peer-review process, which involved up to 20 referees.

Rothwell said that while scientists could often be expected to have different views as to the relative importance of a paper, they should, in theory, be expected to agree about its technical validity - whether an appropriate methodology was used or whether the data presented justified the conclusions - more often.

His study found that journal editors were up to ten times more likely to publish a paper that both referees recommended than if they had disagreed.

Rothwell thought it likely that the personal interests of the most meticulous referee was influencing recommendations.

"The worry some people have about peer review is that it has a built-in bias that tends to maintain the status quo and suppress new theories," he said.

Nevertheless, Rothwell observed that revisions recommended by referees usually improved the quality of the paper and said he could see no obvious alternative way to select papers for publication.

You've reached your article limit.

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Most Commented

James Fryer illustration (27 July 2017)

It is not Luddism to be cautious about destroying an academic publishing industry that has served us well, says Marilyn Deegan

Jeffrey Beall, associate professor and librarian at the University of Colorado Denver

Creator of controversial predatory journals blacklist says some peers are failing to warn of dangers of disreputable publishers

Hand squeezing stress ball
Working 55 hours per week, the loss of research periods, slashed pensions, increased bureaucracy, tiny budgets and declining standards have finally forced Michael Edwards out
Kayaker and jet skiiers

Nazima Kadir’s social circle reveals a range of alternative careers for would-be scholars, and often with better rewards than academia

hole in ground

‘Drastic action’ required to fix multibillion-pound shortfall in Universities Superannuation Scheme, expert warns