Peer review system 'leads to good research going unpublished'

Researchers have used an art competition to model what happens during peer review, and their work raises further questions over its effectiveness

July 26, 2016
Man painting
Source: Alamy

The first experiment to fully model the dynamics of academia’s fiercely competitive peer review system has concluded that it increases creative diversity and innovation, but incentivises researchers to mark their rivals down, meaning that high-quality research goes unpublished.

It adds to doubts over the current system, with one of the authors of the new paper urging journals to experiment further with post-publication peer review, where papers are assessed openly after being made public.

To test what happens when creators assess the work of their rivals – in an attempt to model the peer review system – researchers asked 144 participants in Zurich to come up with works of art on a computer, which were then judged by their fellow participants. This was repeated 30 times, and depending on how well they were rated, participants won up to SFr80 (£61).

Using sliders, participants were able to modify a picture of a face, or transform it beyond recognition into an abstract image. 

Stefano Balietti, a postdoctoral student at the Network Science Institute at Northeastern University in Boston and one of the authors of the paper, said that they “gradually moved towards more abstract art, a bit like what happened in the history of art”.

The researchers then sourced tens of thousands of online reviews of the artworks to come up with as objective a measure of their quality as possible.

They found that when participants were in competition with each other, and had to get high scores from their competitors to win money, they produced more diverse, innovative and abstract work in an attempt to mark themselves out from competitors.

“But more diverse does not mean better,” cautioned Dr Balietti. The experiment also discovered that competition between the participants made them likely to behave “strategically”, marking down rivals to boost their own chances – a common fear about the peer review process.

Competition led to higher rejection rates but did not mean that higher quality art was published, the study found. Dr Balietti said this was most likely because game-playing among participants meant that otherwise good artwork was penalised.

“Our results could explain why many ground-breaking studies in science end up in lower-tier journals,” concludes “Peer review and competition in the Art Exhibition Game”, published in the journal PNAS.

The results feed in to the debate about rejection rates, which at some high-profile journals can reach more than 90 per cent, making competition for inclusion intense.

Dr Balietti said that the results of his experiment indicated that academia faces a trade-off: competition leads to a greater diversity of papers but also means that otherwise good research may not be published.  

When tackling a problem such as climate change, he said, “more competition might not be the best solution. We might by mistake reject the one good solution.”

The paper argues for “career schemes that tolerate early failure and focus on long-term success” as a way to encourage innovation in research. Dr Balietti also argued that more journals should “experiment” with post-publication peer review. 

You've reached your article limit.

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments

Have your say

Log in or register to post comments

Featured Jobs

Most Commented

Daniel Mitchell illustration (29 June 2017)

Academics who think they can do the work of professional staff better than professional staff themselves are not showing the kind of respect they expect from others

celebrate, cheer, tef results

Emilie Murphy calls on those who challenged the teaching excellence framework methodology in the past to stop sharing their university ratings with pride

Sir Christopher Snowden, former Universities UK president, attacks ratings in wake of Southampton’s bronze award

Senior academics at Teesside University put at risk of redundancy as summer break gets under way

Tef, results, gold, silver, bronze, teaching excellence framework

The results of the 2017 teaching excellence framework in full. Find out which universities were awarded gold, silver or bronze