Peer review system 'leads to good research going unpublished'

Researchers have used an art competition to model what happens during peer review, and their work raises further questions over its effectiveness

July 26, 2016
Man painting
Source: Alamy

The first experiment to fully model the dynamics of academia’s fiercely competitive peer review system has concluded that it increases creative diversity and innovation, but incentivises researchers to mark their rivals down, meaning that high-quality research goes unpublished.

It adds to doubts over the current system, with one of the authors of the new paper urging journals to experiment further with post-publication peer review, where papers are assessed openly after being made public.

To test what happens when creators assess the work of their rivals – in an attempt to model the peer review system – researchers asked 144 participants in Zurich to come up with works of art on a computer, which were then judged by their fellow participants. This was repeated 30 times, and depending on how well they were rated, participants won up to SFr80 (£61).

Using sliders, participants were able to modify a picture of a face, or transform it beyond recognition into an abstract image. 

Stefano Balietti, a postdoctoral student at the Network Science Institute at Northeastern University in Boston and one of the authors of the paper, said that they “gradually moved towards more abstract art, a bit like what happened in the history of art”.

The researchers then sourced tens of thousands of online reviews of the artworks to come up with as objective a measure of their quality as possible.

They found that when participants were in competition with each other, and had to get high scores from their competitors to win money, they produced more diverse, innovative and abstract work in an attempt to mark themselves out from competitors.

“But more diverse does not mean better,” cautioned Dr Balietti. The experiment also discovered that competition between the participants made them likely to behave “strategically”, marking down rivals to boost their own chances – a common fear about the peer review process.

Competition led to higher rejection rates but did not mean that higher quality art was published, the study found. Dr Balietti said this was most likely because game-playing among participants meant that otherwise good artwork was penalised.

“Our results could explain why many ground-breaking studies in science end up in lower-tier journals,” concludes “Peer review and competition in the Art Exhibition Game”, published in the journal PNAS.

The results feed in to the debate about rejection rates, which at some high-profile journals can reach more than 90 per cent, making competition for inclusion intense.

Dr Balietti said that the results of his experiment indicated that academia faces a trade-off: competition leads to a greater diversity of papers but also means that otherwise good research may not be published.  

When tackling a problem such as climate change, he said, “more competition might not be the best solution. We might by mistake reject the one good solution.”

The paper argues for “career schemes that tolerate early failure and focus on long-term success” as a way to encourage innovation in research. Dr Balietti also argued that more journals should “experiment” with post-publication peer review.

david.matthews@tesglobal.com 

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Most Commented

question marks PhD study

Selecting the right doctorate is crucial for success. Robert MacIntosh and Kevin O'Gorman share top 10 tips on how to pick a PhD

India, UK, flag

Sir Keith Burnett reflects on what he learned about international students while in India with the UK prime minister

Pencil lying on open diary

Requesting a log of daily activity means that trust between the institution and the scholar has broken down, says Toby Miller

Application for graduate job
Universities producing the most employable graduates have been ranked by companies around the world in the Global University Employability Ranking 2016
Construction workers erecting barriers

Directly linking non-EU recruitment to award levels in teaching assessment has also been under consideration, sources suggest