Reviewers change their mind after hearing from ‘peers’

Harvard experiment finds nearly half of academics amended their assessment of a research application after seeing randomly generated ‘expert scores’

May 8, 2019
Whisper in ear

Any academic who has submitted a research proposal or a journal paper will have felt the sting of an unwelcome comment from at least one peer reviewer – inevitably leading to grumbles among friends that said reviewer simply doesn’t know what they’re talking about.

Now a study by Harvard University researchers offers evidence to suggest that almost half of academic reviewers actually share that lack of confidence in their own expertise.

For the experiment, outlined in a working paper published by Harvard Business School, researchers recruited 277 academics from seven different US medical schools to assess funding applications. After evaluating a proposal, the scholars were shown a range of ratings and told that they had been given to the same application by experts in their field, although they had in fact been randomly assigned to be above or below a reviewer’s initial mark.

Having seen this, nearly half of reviewers – 47.1 per cent – chose to amend their rating.

The exercise was anonymised, and a control experiment in which reviewers were not shown others’ scores but were given the option to amend their own resulted in no changes, demonstrating that external influences were the key prompt for scoring revisions.

Significantly, women taking part in the experiment were 13.9 per cent more likely than men to update their review scores. Women working in male-dominated subfields were particularly susceptible to the influence of others, the study found.

Meanwhile, very highly cited “superstar” reviewers – who were more likely to be male – changed their scores 24 per cent less often than others.

Misha Teplitskiy, a postdoctoral fellow at Harvard’s Laboratory for Innovation Science and co-author of the study, said that willingness to change one’s views was not a bad thing in itself. “In fact, it’s generally good to take other opinions into account, all the more so when those opinions come from serious experts,” he told Times Higher Education. What was concerning, however, was that certain groups revised their scores more than others, “even when they have similar expertise”.

In all but one case, reviewers revised their scores towards those given by others. Reviewers who did change their own scores were most likely to do so by one point (86.6 per cent), but researchers cautioned that these “seemingly small updates can have dramatic implications for funding outcomes”.

The difference in self-confidence levels also has wider implications, the report concludes, because “individuals who underrate their competence may be less likely to apply for grants, ask for resources, or seek recognition for their achievements”.

rachael.pells@timeshighereducation.com

POSTSCRIPT:

Print headline: Peek at peers’ points changes reviewers’ minds

Please login or register to read this article.

Register to continue

Get a month's unlimited access to THE content online. Just register and complete your career summary.

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Related articles

Peer review is lauded in principle as the guarantor of quality in academic publishing and grant distribution. But its practice is often loathed by those on the receiving end. Here, seven academics offer their tips on good refereeing, and reflect on how it may change in the years to come

6 December

Related universities

Have your say

Log in or register to post comments

Most commented

Sponsored