Peer review system 'leads to good research going unpublished'

Researchers have used an art competition to model what happens during peer review, and their work raises further questions over its effectiveness

July 26, 2016
Man painting
Source: Alamy

The first experiment to fully model the dynamics of academia’s fiercely competitive peer review system has concluded that it increases creative diversity and innovation, but incentivises researchers to mark their rivals down, meaning that high-quality research goes unpublished.

It adds to doubts over the current system, with one of the authors of the new paper urging journals to experiment further with post-publication peer review, where papers are assessed openly after being made public.

To test what happens when creators assess the work of their rivals – in an attempt to model the peer review system – researchers asked 144 participants in Zurich to come up with works of art on a computer, which were then judged by their fellow participants. This was repeated 30 times, and depending on how well they were rated, participants won up to SFr80 (£61).

Using sliders, participants were able to modify a picture of a face, or transform it beyond recognition into an abstract image. 

Stefano Balietti, a postdoctoral student at the Network Science Institute at Northeastern University in Boston and one of the authors of the paper, said that they “gradually moved towards more abstract art, a bit like what happened in the history of art”.

The researchers then sourced tens of thousands of online reviews of the artworks to come up with as objective a measure of their quality as possible.

They found that when participants were in competition with each other, and had to get high scores from their competitors to win money, they produced more diverse, innovative and abstract work in an attempt to mark themselves out from competitors.

“But more diverse does not mean better,” cautioned Dr Balietti. The experiment also discovered that competition between the participants made them likely to behave “strategically”, marking down rivals to boost their own chances – a common fear about the peer review process.

Competition led to higher rejection rates but did not mean that higher quality art was published, the study found. Dr Balietti said this was most likely because game-playing among participants meant that otherwise good artwork was penalised.

“Our results could explain why many ground-breaking studies in science end up in lower-tier journals,” concludes “Peer review and competition in the Art Exhibition Game”, published in the journal PNAS.

The results feed in to the debate about rejection rates, which at some high-profile journals can reach more than 90 per cent, making competition for inclusion intense.

Dr Balietti said that the results of his experiment indicated that academia faces a trade-off: competition leads to a greater diversity of papers but also means that otherwise good research may not be published.  

When tackling a problem such as climate change, he said, “more competition might not be the best solution. We might by mistake reject the one good solution.”

The paper argues for “career schemes that tolerate early failure and focus on long-term success” as a way to encourage innovation in research. Dr Balietti also argued that more journals should “experiment” with post-publication peer review.

david.matthews@tesglobal.com 

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Featured Jobs

Assistant Professorship in Behavioural Science LONDON SCHOOL OF ECONOMICS & POLITICAL SCIENCE LSE
Foundation Partnerships Officer LONDON SCHOOL OF ECONOMICS & POLITICAL SCIENCE LSE

Most Commented

Social media icons

Gabriel Egan laments the narcissistic craving for others’ approval brought on, he says, by the use of social networking websites

James Fryer illustration (8 September 2016)

Some lecturers will rightly encourage forms of student interaction that are impossible for those covering their faces, Eric Heinze argues

Elly Walton illustration (25 August 2016)

Treating students as consumers has precipitated a rush to the bottom to give them exactly what they want, says John Warren

Handwritten essay on table

Universities must pay more attention to the difficulties faced by students, says Daniel Dennehy

Theresa May entering 10 Downing Street, London

The prospect of new grammar schools on the horizon raises big questions for HE, writes Nick Hillman