Peer review not yet sunk

June 4, 2015

Richard Smith says that when he was editor of the British Medical Journal, he deliberately inserted eight errors into a paper and sent it to 300 reviewers. None of them spotted more than five errors (“Ineffective at any dose? Why peer review simply doesn’t work”, Opinion, 28 May).

I was one of those reviewers, unaware at the time that this particular paper was part of an experiment. I sent a covering letter with my review that went something like this: “This paper is not only unpublishable in its present form; it contains a number of fatal flaws that suggest it will never be publishable. I list below three of those fatal flaws. If you would like me to take a closer look at the paper to provide a more comprehensive set of feedback for the authors, please let me know.”

The BMJ’s staff – then as now – viewed peer review as a technical task (“spotting errors”) rather than a scholarly one (interpretation and judgement). They never did ask me for a more detailed list of errors. Presumably, taking account of authors’ covering letters was not part of their study protocol. I am sure many reviewers would have done what I did: fire a quick paragraph to hole the paper below the waterline and return to their day job. Smith’s conclusion is therefore not the only one that can be drawn from his dataset.

Like all experiments, artificial studies of peer review strip the process of its social context. They contain the ossified assumptions of the study designers – and for that reason they are biased in a way that will tend to confirm the authors’ initial hypothesis.

Trisha Greenhalgh
Professor of primary care health sciences
University of Oxford

Richard Smith’s argument about the ineffectiveness of peer review is weakened rather than strengthened by this claim: “With the World Wide Web everything can be published, and the world can decide what’s important and what isn’t.”

Peer review is flawed, but it is naive to claim that the internet alone will bring scientists together to present and critique their studies. This is demonstrated by the arXiv preprint server for physics and mathematics research. arXiv is not peer reviewed. There is good and bad there, but one must know enough about the subject matter to distinguish between the two.

The root of the problem is ethical integrity. Without honesty, neither peer review nor online vetting will be effective.

Lisa E. Wells

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments

Have your say

Log in or register to post comments

Featured Jobs

Dental Clinical Skills Assistant UNIVERSITY OF BRISTOL

Most Commented

question marks PhD study

Selecting the right doctorate is crucial for success. Robert MacIntosh and Kevin O'Gorman share top 10 tips on how to pick a PhD

India, UK, flag

Sir Keith Burnett reflects on what he learned about international students while in India with the UK prime minister

Pencil lying on open diary

Requesting a log of daily activity means that trust between the institution and the scholar has broken down, says Toby Miller

Application for graduate job
Universities producing the most employable graduates have been ranked by companies around the world in the Global University Employability Ranking 2016
Construction workers erecting barriers

Directly linking non-EU recruitment to award levels in teaching assessment has also been under consideration, sources suggest