Take the rough with the smooth

Adrian Furnham has had his share of peer review nightmares, but the frailties of the system have also worked in his favour

May 23, 2013

Source: Patrick Wellham

In the past couple of months, Times Higher Education has highlighted two examples of what would appear to be flagrant abuses of the peer review system.

One story concerned a journal editor who deleted positive comments from several referees’ reports in an apparent attempt to justify her decision to reject a manuscript.

The other story described the decision by the Committee on Publication Ethics - after some debate - to condemn editors who secretly write their own reviews when they can’t find enough referees. Much of the reaction to the story amounted to shock that any editor would do such a thing.

Having published more than 1,000 peer-reviewed papers (and kept all the reviews), I can report four natural - if unintentional - experiments that lend further weight to concerns about the apparent arbitrariness of journal decision-making.

The most interesting concerns a well-known US-based journal, to which I submitted a manuscript a few years ago. Some months later the editor recommended “revise and resubmit” based on three thoughtful reviews all recommending different changes. I did my best to address the concerns and, after another round of changes, the paper was accepted.

Two weeks after I was sent the proofs, I received another letter from the same editor enclosing two new reviews of my manuscript: one curtly dismissive and the other lukewarm. On this basis, he had decided to reject it. Was this a short-term memory problem, overwork or a prank, I wondered. When I pointed out that he had already accepted the manuscript, the editor was mortified and we agreed to ignore the second set of reviews and forget the “whole embarrassing episode”. The paper, by the way, has received what I would deem to be modest citations over the years.

Curiously, the electronification of the submission process seems to have made it even more chaotic. Over the past three years, two other journals have processed one of my manuscripts twice, using different sets of reviewers. In both instances, one response letter contained a flat rejection, while the other contained a rather positive “revise and resubmit”. In one case, I responded only to the positive letter, revised the manuscript as recommended and had the paper accepted. In the other case, I pointed out what had happened - prompting an apology and a rejection from the editor. That manuscript was later accepted by a journal with a higher impact factor.

In the final inadvertent experiment, the editor concerned falsely assumed that my manuscript had been reviewed and revised, so accepted it as submitted. In the interests of science I said nothing and have been checking the citation scores ever since. They are modest but acceptable.

I also have a letter with about three mistakes per sentence complaining about my own typographical errors. Another editor suggested that I give my paper to a fluent English speaker for help; this was on the very day I had my 400th column published in a top British newspaper.

Not that any of this should come as a great surprise. As long ago as 1982, a famous paper by Douglas Peters and Stephen Ceci described how most papers resubmitted, as an experiment, to the same prestigious psychology journals that had originally published them were rejected the second time round. There is now a whole literature on the topic.

I tell postgraduates just starting out on their publish-or-perish journeys how it all works. I tell them how difficult it is for editors to get anyone to review a paper; how a negative review has the power to “blackball” all others; and how sometimes you get rejected because you have not quoted the editor or reviewers enough.

I also tell them about a great editor I knew who told me to ignore certain reviewers’ comments and who had been known to accept papers when both reviewers had recommended rejection, and vice versa. That is an editor’s job - but most seem always to concur with reviewers, acting more in a secretarial than a judgemental role.

Editors are often too timid to tell reviewers when they do a bad job (because they are doing it for free) and too lazy to strike those individuals from their list. Hence, some reviewers are permitted to get away with gratuitous nastiness in pursuit of personal vendettas, or with submitting negative reviews simply because they pride themselves on never “accepting” manuscripts.

And editors themselves aren’t above abusing their corrupting, godlike power, looking after their friends, punishing their enemies and propagating their pet theories. But even for those who are conscientious, being a journal editor is not unlike being a radiologist who makes a diagnosis but only rarely gets feedback on its accuracy. In such circumstances, it is really tough to learn from your mistakes even if you want to.

But I also tell my students that coping with rejection is at the heart of the academic enterprise, and to take heart. After all, no one has come up with a better system than peer review. And, as in some of the examples I’ve given, its frailties can sometimes work for you as well as against you.

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Reader's comments (1)

This is a great article from the point of view of someone whose been "in the trenches". Refreshingly so, as so much of the commentary on this seems to come from the non-practitioner side. That said, I have to take issue with the concluding paragraph, because "no one has come up with a better system" seems to suggest that everything is OK, when it's clearly not. Peer review alone does not do the job of adequately filtering the literature, as documented by Glenn Begley and others: http://www.reuters.com/article/2012/03/28/us-science-cancer-idUSBRE82R12P20120328 In addition, peer review is broadening beyond just three reviewers and an editor making a pre-publication decision. Jason Priem has a great paper on this topic here: http://www.nature.com/nature/journal/v495/n7442/full/495437a.html

Have your say

Log in or register to post comments

Most Commented

question marks PhD study

Selecting the right doctorate is crucial for success. Robert MacIntosh and Kevin O'Gorman share top 10 tips on how to pick a PhD

Pencil lying on open diary

Requesting a log of daily activity means that trust between the institution and the scholar has broken down, says Toby Miller

India, UK, flag

Sir Keith Burnett reflects on what he learned about international students while in India with the UK prime minister

Application for graduate job
Universities producing the most employable graduates have been ranked by companies around the world in the Global University Employability Ranking 2016
Construction workers erecting barriers

Directly linking non-EU recruitment to award levels in teaching assessment has also been under consideration, sources suggest