Should post-publication peer review be anonymous?

The PubPeer organisers and two other scholars debate the merits and pitfalls of anonymity in peer review

December 10, 2015
Henny Porten as Anne Boleyn, 1920
Source: Corbis

Among the criticisms that are frequently made of peer review is that referees’ anonymity allows laziness, discourtesy and conflicts of interest to flourish. Editors do their best to broker honestly, but grievances were not in short supply when Times Higher Education asked around earlier this year for particularly egregious examples of reviewer comments that people had received (“The worst piece of peer review I’ve ever received”, Features, 6 August).

There is even greater concern about anonymity in the case of so-called post-publication peer review (PPPR), which permits anyone with an internet connection to comment on published papers. In October, the journal Plant Physiology published a scathing editorial stating that contributions to the largely anonymous PPPR forum PubPeer often lacked “courtesy and common sense”. The article, written by the journal’s editor-in-chief Michael Blatt, Regius professor of botany at the University of Glasgow, said: “Given that the majority of comments show the most petty kind of scientific criticism, can there be any doubt that the intent often is to pillory, to do so publicly and without accountability?”

PubPeer’s founders were themselves anonymous until August, when they revealed their identity as part of an application to establish the site as a non-profit organisation. Here, in an exchange of letters with Philip Moriarty, professor of physics at the University of Nottingham, they set out their case for commenter anonymity, while Moriarty counters that Blatt is right to worry about the tone that this lends to discussion.

Meanwhile, Paul Benneworth, a principal researcher at the University of Twente in the Netherlands, argues that anonymity remains crucial to all forms of review – and outs himself as the probable author of the offending referee’s report highlighted by one of the contributors to THE’s article. Publishing may be in an era of increasing openness and transparency, but it is clear that the case for abandoning anonymity in research is far from clear-cut.


Dear Philip,

There is a growing awareness that a large proportion of published scientific research is of low quality and is unreproducible, with the problem being acute in life and social sciences. This has potentially enormous economic and human costs. Using taxpayers’ money to produce and build on unreliable research is wasteful, and researchers’ attempts to extend unreliable research will be unproductive and may damage their careers. Public policy or medical guidelines based upon flawed research may cause further waste and even cost lives.

Science is said to be self-correcting, but within the traditional publishing system, effective correction requires another publication. Such “rebuttals” are often discouraged by journals’ editorial guidelines. Even when they are not, writing and publishing a whole new paper via traditional publishing is often time-consuming, onerous, expensive and excruciatingly slow. Internet technologies can enable instant, widespread dissemination of any information relevant to a publication, and there is a broad consensus that this would offer a huge improvement over the current system. The only real point of dispute is whether anonymous commenting on published scientific work should be banned.

The US Constitution provides strong protection for anonymous speech in order, according to the US Supreme Court, to “protect unpopular individuals from retaliation” (McIntyre v. Ohio Elections Commission, 1995). But it is not an absolute right. How a community regulates anonymity typically hinges upon practical and moral considerations. In the absence of an overriding imperative, the decision to allow anonymous scientific discussion on PubPeer is based upon our careful balancing of the risks and benefits of doing so.

Our experience has been that anonymity greatly facilitates scientific discussion with little if any effect on the quality of comments. The success and growth of PubPeer (we receive approximately 1,000 comments and 500,000 page views a month) can be compared with the activity on other services that forbid anonymity: the absolutely moribund commenting facilities provided by journals and the relatively steady, low-level use of the PubMed Commons system that is offered by the PubMed biomedical archive. In addition, we find that without anonymity, many serious issues are simply never aired – presumably because the competitive and hierarchical environment in research causes most researchers to avoid public criticism of colleagues.

The main arguments against anonymity are that it allows commenters to blacken the names of their competitors with impunity and to hide conflicts of interest. Again, our practical experience has been that these issues rarely arise, for several reasons. First, comments on PubPeer must be based upon verifiable information (usually the data in a publication), which focuses discussion on the science. We moderate comments and provide reporting facilities to enforce this guideline. Second, only cogent scientific arguments will convince readers. Third, authors are notified of comments by email and are encouraged to explain or defend their work; many now do so. (Incidentally, most open questions on PubPeer would be resolved instantly by access to the raw data described in the publication, so we can expect such cases to decline with the trend towards data sharing.)

A weaker argument against anonymity is more subjective: the world would be a better place if everybody signed their comments. We agree. But our experience today shows that a critical mass of comments cannot be achieved without anonymity. Faced with this reality, PubPeer has chosen a commenting policy that is aimed at maximising the dissemination of relevant information and reducing the economic and human waste from unreliable research, at the potential cost of a few ruffled academic feathers.

Yours,
The PubPeer organisers


Dear PubPeer organisers,

I am an enthusiastic supporter of PPPR. A publication should represent the start, not the end, of scientific discussion and debate. I also regularly sing the praises of PubPeer, and your comments regarding open access to raw data also resonate strongly with me.

I would quibble, however, with your claim that there is a “broad consensus” that online PPPR is a welcome change. Very many academics and publishers see it as rather more of a significant threat than an opportunity. An editorial in the American Chemical Society’s Nano journal a couple of years back forcefully made this point. “When plagiarism or data manipulation is suspected, accusations should be reported directly to the journal in which the article is published. It is best not to make an accusation anonymously,” it said.

I disagree entirely that the most appropriate channel for scientific critique is a protracted and opaque letter-to-the-editor approach; my colleagues and I know from bitter experience just how ineffectual that process can be. And we also know just how essential PubPeer can be in highlighting questionable research.

But my views diverge significantly from yours when it comes to the ACS Nano editors’ misgivings – repeated in the recent Plant Physiology editorial – on the central role that anonymity plays in PPPR. I am firmly of the opinion that, too often, online anonymity debases communication and that it is impeding the wider acceptance of PPPR as the natural platform for scientific critique, criticism and debate.

You say that the “only real point of dispute” is whether anonymous commenting should be banned. This, however, is not at all representative of my position. I want to strongly discourage anonymous commenting, to attempt to change the commenting culture so that anonymity is not the norm. But I do not want to ban it. (Just how could one realistically ban it in any case?)

You admit that “the world would be a better place if everybody signed their comments” but you are resigned to accepting anonymous commenting as the norm because “the competitive and hierarchical environment in research causes most researchers to avoid public criticism of colleagues”. This is a shocking indictment of our research system; it’s a timid, rather grubby and damaging mindset that has no place in science. As my colleague Julian Stirling, a research fellow at the National Institute of Standards and Technology in Maryland, once put it during a lengthy online debate: “If you plan to get to the top of your discipline without ever publicly questioning established views, then you have a very strange view of science.”

Yours,
Philip Moriarty


Dear Philip,

Your response indicates that our positions are in fact quite close: you support post-publication peer review and recognise that anonymous commenting is here to stay. Our main disagreement is over the degree to which that anonymity should be encouraged or discouraged. Our decision at PubPeer has been to focus on the content of comments. If that is factual and relevant, then anonymity is acceptable, especially as it has in practice greatly encouraged information-sharing. In our experience, there is little truth in the assertion sometimes made that anonymous comments are of lower quality.

Like many other professional fields, science is competitive and hierarchical. In such systems, it is not surprising that the less established in particular may worry about retaliation for voicing criticism. We agree with you that the fear of retribution implied by the current desire for anonymity is a “shocking indictment of our research system”, but we blame the system, not the commenters.

We are hopeful that services such as PubPeer will slowly tip the balance towards the acceptance of open discussion and criticism. For this to happen, the research community will need to stop turning a blind eye to poor-quality work and become much more supportive of those who raise valid concerns. In the meantime, we think it is unfair of opponents of anonymity to demand that PhD students, postdocs or indeed any other scientists take risks that most established researchers, journals and institutions currently avoid. At PubPeer, we do not consider those who choose to share their expertise anonymously to be “grubby and damaging”.

Yours,
The PubPeer organisers


Dear PubPeer organisers,

Our positions on PPPR are indeed very close, but I remain soundly of the opinion that, too often, anonymity debases debate. You argue that if comments are “factual and relevant” then “anonymity is acceptable”. But acceptable to whom?

In the lengthy (and sometimes rather overheated) PubPeer thread that dissects the Plant Physiology editorial, one of the contributors draws a rather telling comparison between online anonymity and cold-calling. Would you find it acceptable to receive a phone call from an individual who refused to tell you their name, or anything about their background, but criticised your most recent paper at length?

I’ll again stress that this is not a question of the quality of the comments – I will agree that often, but certainly not invariably, anonymous comments are scientifically sound. My key concerns about anonymity are expressed much more eloquently than I could manage by Blatt himself: “Anonymity is intimidating in itself, regardless of how polite a commenter may be; psychologically, it wrong-foots the author from the start, and the price often is an absence of worthwhile discussion.” This asymmetry in communication is key.

You’ve argued in the past that anonymity counters the (greater?) asymmetry that exists between early career researchers who challenge published research and those prestigious professors whose work is being criticised. While I have quite some sympathy for your stance, it is nonetheless the case that anonymity is now the norm in PPPR, regardless of the extent to which career “fallout” might be a possibility. The PubPeer thread on the Blatt editorial is a great example of this – the vast majority of those commenting are anonymous, yet their comments are innocuous in the context of any type of potential damage to their careers. I will always challenge that norm.

Yours,
Philip Moriarty

Adam West as Batman, close-up
Source: 
Corbis

In praise of harsh reviews

I read the THE feature highlighting supposedly egregious peer reviews with a growing sense of unease. One anonymous contributor recalled a referee who had criticised his paper for its supposedly poor conceptual awareness, “sub-tabloid” writing style and “abysmal” grammar, likening it to a mid-level undergraduate essay. The feature then stated that the paper had subsequently received plaudits after being published elsewhere, inviting us to conclude that traditional, “blind” peer review has descended into a snake pit of personal abuse, petty jealousies and pet theories, and no longer serves as a dispassionate system of scientific judgement.

An increasingly common view seems to be that if referees were obliged to sign their reviews, this would force them to offer more temperate, considered and constructive criticism, and that science would be all the better for it. As an editor of Regional Studies, Regional Science, a journal that proudly offers early career authors a mentored publication route to short-circuit these issues, I’m well aware of the pitfalls of standard peer review. But I would also argue that the option to write harsh reviews is central to any thriving science system that prizes integrity over system-gaming.

I must here declare an interest, because the comments that the THE contributor received may very well have come from me; I have certainly written a handful of similarly harsh reviews in my time. But I’ve also received two particularly scathing reviews myself that stuck in my memory. The first was on a paper I wrote as a postdoc: it said that my whole premise was based on my excitement with a literature that was probably new only to me. The second was an eight-page line-by-line demolition of a qualitative paper by a reviewer who apparently wanted quantitative tests of statistical significance.

On each occasion, I was stunned at first, but I have to concede that, on reflection, the reviewers were almost certainly right. Both papers were submitted to leading journals that favour a very particular style of article, and my research was not a comfortable fit in either. Without those harsh reviews, the editors might have been tempted to give me a chance to squeeze my findings into their journals’ style. And with the lure of a leading publication, the younger me might have been tempted to “play the game” and say something other than what my findings would strictly have warranted.

Many advocates of open review believe that it should be carried out post-publication. But if that model took off, I predict that a new class of specialist “star reviewers” would emerge. They would acquire the power to make or break papers in the same way that influential bloggers can make or break new products, and authors and editors would respond by mechanically writing and selecting only those papers that they believed would attract the star reviewers’ praise. This is hardly a recipe for fostering the creation of challenging new ideas.

Meanwhile, if open reviewing were carried out pre-publication, as it already is in some journals, reviewers would inevitably be inclined to shy away from negative comments for fear of upsetting authors who are also their potential future referees. Either way, publishing would become a little bit less honest – and it would be science and scientists that lost out.

Paul Benneworth is a principal researcher in the Center for Higher Education Policy Studies at the University of Twente in the Netherlands.

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

POSTSCRIPT:

Print headline: What's in a name?

Reader's comments (3)

The argument about the utility (NOT the value) of anonymous versus signed comments could easily be put to the test by inviting people to sign if they wish/or not and see how many choose the option. Of course PubPeer commentators can, surely, include their name in the comment anyway. The value of the enterprise simple cannot be gauged at this stage; it is too early. We need to wait to see how many retractions, errata and corrigenda can be attributed to PubPeer or, at the sharp end, how many lives saved, for example, in medical research. However it operates and how effective it is, the procedure of not going to journal editors with queries about articles is not what the Committee on Publication Ethics (COPE) recommends and the editors of most journals are signed up to COPE. If I saw an anonymous comment about an article in my journal (Journal of Advanced Nursing) on PubPeer I would be less inclined to act than if someone had followed the COPE guidance. As it stands there are comments on two articles in my journal on PubPeer: the oldest single comment takes you to a blog on the issue discussed in the paper - someone drawing attention to their blog and, therefore, once at the blog, not anonymous. As for the second set of comments, one is a warning about some of the literature in the field of the article and the other seems to be the search strategy cut and pasted from a paper - but not sure if it is the one in my journal (not checked yet as that is not the point) and, while it is clear which comment was posted first (the search strategy), it is not clear what point it is trying to make or how the second post addresses that point. So, not sure what value this is to anyone.
The Introduction to this exchange references the editorial by Michael Blatt stating that contributions to PubPeer 'often lacked “courtesy and common sense”. ' That editorial, in my opinion, lacked courtesy. But the editor-in-chief of a respected journal, with a named professorship, is not likely to face significant consequences for his lack of courtesy. Those of use lower in status who publicly criticize (however politely and reasonably) the science of those high in status, do fear significant consequences. We fear that we might face reprisals in reviews of our applications for grants and fellowships, or of our manuscript submissions. I suspect that many of those who are high in status do not understand how others react to them.
On pitfalls of anonymity: I am independent science journalist and have recently blogged about the anonymity of post-publication peer review (PPPR). There, I present two real-life examples on 1) how anonymous PPPR can be abused 2) how signed PPPR can achieve results in correcting scientific literature. Your comments are welcome: https://forbetterscience.wordpress.com/2015/12/13/post-publication-peer-review-signed-or-anonymous/

Have your say

Log in or register to post comments

Most Commented

question marks PhD study

Selecting the right doctorate is crucial for success. Robert MacIntosh and Kevin O'Gorman share top 10 tips on how to pick a PhD

Pencil lying on open diary

Requesting a log of daily activity means that trust between the institution and the scholar has broken down, says Toby Miller

India, UK, flag

Sir Keith Burnett reflects on what he learned about international students while in India with the UK prime minister

Application for graduate job
Universities producing the most employable graduates have been ranked by companies around the world in the Global University Employability Ranking 2016
Construction workers erecting barriers

Directly linking non-EU recruitment to award levels in teaching assessment has also been under consideration, sources suggest