Life sciences' REF triumph 'lacks credibility'

Bibliometric analysis concludes lower-quality thresholds in 2014 are more likely explanation for doubling of top-rated research

July 27, 2015
Tarsier has eyes that are larger than it's brain. Bohol island, Philippines
Source: Alamy
Stardust: 4* work doubled in six years

The implication of the research excellence framework results that the amount of world-leading research carried out in the UK has doubled in the past six years “lacks credibility”.

This is the claim of a team led by Jonathan Grant, director of the Policy Institute at King’s College London, which used bibliometrics to examine the apparently “remarkable improvement” in UK life science research that saw a doubling in the proportion of work graded 4* (“world-leading”) in the 2014 REF, compared with the 2008 research assessment exercise (rising from 11.8 to 23.9 per cent of the total).

The team found an increase of only 25 per cent in the proportion of submitted papers that were in the top 10 per cent of global papers published in the same field in the same year. Since 9 per cent fewer life science articles were submitted to the REF, the absolute number of papers in the top 10 per cent was just 10 per cent higher.

The results are recorded in a paper, “UK doubles its ‘world-leading’ research in life sciences and medicine in six years: testing the claim?”, published in Plos One on 23 July.

The number of life sciences articles rated 4* in the RAE was the same as the number of submitted papers that were in the top 4.4 per cent of global papers ranked by citation impact. That figure was 7.3 per cent for the REF. This, according to the paper, suggests that the main reason for grade inflation in the REF was that its panels used “a somewhat lower threshold of acceptance for a 4* level”.

By comparison, the difference between thresholds in the physical sciences was less marked. The number of such papers rated 4* in the RAE was the same as the number in the top 6.5 per cent globally, compared with 7.7 per cent in the REF.

Thresholds may have been lowered, the paper suggests, due to the increased reward in 2014 for 4* papers in the formula allocating quality-related research funding. Since the REF results “have huge implications for the relative standing of fields, the research funding of universities, and funding allocated within universities to different research groups” it is “understandable…the REF evaluations are likely to be influenced by these external factors”.

Metrics are, of course, controversial. But the paper argues that, especially at a large scale, they can “provide an external indicator of research quality”.

“For the advancement of science and health, evaluation of research quality requires consistency and validity [and] the discrepancy noted here calls for a closer examination of mass peer-review methods like the REF,” it adds.

Speaking to Times Higher Education, Professor Grant added that grade inflation “makes it harder to distinguish between different degrees of ‘excellence’ within the 4* category”.

This “potentially means the performance-related funding aspects are undermined and the incentive to grow true quality is diluted”, he said.

“Being in the top 4.4 or 7.3 per cent [by citation impact] is clearly very high performance…but to claim that the amount of ‘world-leading’ research has doubled over that period lacks credibility and validity internationally.”

Please login or register to read this article

Register to continue

Get a month's unlimited access to THE content online. Just register and complete your career summary.

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments

Related articles


Print headline: REF triumph of life sciences ‘lacks credibility’

Have your say

Log in or register to post comments