A very random peer review

The peer review, one of the pillars of modern scientific method, is a perilously random process, according to a new study.

Researchers have carried out detailed analysis of how the opinions of referees used by two leading clinical neuroscience journals tallied with one another, and how this influenced whether a scientific paper was accepted for publication. They also looked at the submission process for two conferences.

Peter Rothwell, a neurologist at the Radcliffe Infirmary, Oxford, and Christopher Martyn, at the Medical Research Council's environmental epidemiology unit, Southampton, found that the incidence of agreement between the expert reviewers as to whether a paper should be accepted for publication, rejected or revised, was not statistically significant.

Rothwell said: "If you gave two chimpanzees pens and got them to stick them into one paper or the other, they'd agree about half the time as well."

The research, published in the latest issue of the journal Brain, backs up work that found similar problems with the peer-review process in fields such as psychology and physical sciences.

The experts compared comments from pairs of independent referees on 179 submitted papers to one journal and 116 to another. They also looked at the conference peer-review process, which involved up to 20 referees.

Rothwell said that while scientists could often be expected to have different views as to the relative importance of a paper, they should, in theory, be expected to agree about its technical validity - whether an appropriate methodology was used or whether the data presented justified the conclusions - more often.

His study found that journal editors were up to ten times more likely to publish a paper that both referees recommended than if they had disagreed.

Rothwell thought it likely that the personal interests of the most meticulous referee was influencing recommendations.

"The worry some people have about peer review is that it has a built-in bias that tends to maintain the status quo and suppress new theories," he said.

Nevertheless, Rothwell observed that revisions recommended by referees usually improved the quality of the paper and said he could see no obvious alternative way to select papers for publication.