Life sciences' REF triumph 'lacks credibility'

Bibliometric analysis concludes lower-quality thresholds in 2014 are more likely explanation for doubling of top-rated research

July 27, 2015
Tarsier has eyes that are larger than it's brain. Bohol island, Philippines
Source: Alamy
Stardust: 4* work doubled in six years

The implication of the research excellence framework results that the amount of world-leading research carried out in the UK has doubled in the past six years “lacks credibility”.

This is the claim of a team led by Jonathan Grant, director of the Policy Institute at King’s College London, which used bibliometrics to examine the apparently “remarkable improvement” in UK life science research that saw a doubling in the proportion of work graded 4* (“world-leading”) in the 2014 REF, compared with the 2008 research assessment exercise (rising from 11.8 to 23.9 per cent of the total).

The team found an increase of only 25 per cent in the proportion of submitted papers that were in the top 10 per cent of global papers published in the same field in the same year. Since 9 per cent fewer life science articles were submitted to the REF, the absolute number of papers in the top 10 per cent was just 10 per cent higher.

The results are recorded in a paper, “UK doubles its ‘world-leading’ research in life sciences and medicine in six years: testing the claim?”, published in Plos One on 23 July.

The number of life sciences articles rated 4* in the RAE was the same as the number of submitted papers that were in the top 4.4 per cent of global papers ranked by citation impact. That figure was 7.3 per cent for the REF. This, according to the paper, suggests that the main reason for grade inflation in the REF was that its panels used “a somewhat lower threshold of acceptance for a 4* level”.

By comparison, the difference between thresholds in the physical sciences was less marked. The number of such papers rated 4* in the RAE was the same as the number in the top 6.5 per cent globally, compared with 7.7 per cent in the REF.

Thresholds may have been lowered, the paper suggests, due to the increased reward in 2014 for 4* papers in the formula allocating quality-related research funding. Since the REF results “have huge implications for the relative standing of fields, the research funding of universities, and funding allocated within universities to different research groups” it is “understandable…the REF evaluations are likely to be influenced by these external factors”.

Metrics are, of course, controversial. But the paper argues that, especially at a large scale, they can “provide an external indicator of research quality”.

“For the advancement of science and health, evaluation of research quality requires consistency and validity [and] the discrepancy noted here calls for a closer examination of mass peer-review methods like the REF,” it adds.

Speaking to Times Higher Education, Professor Grant added that grade inflation “makes it harder to distinguish between different degrees of ‘excellence’ within the 4* category”.

This “potentially means the performance-related funding aspects are undermined and the incentive to grow true quality is diluted”, he said.

“Being in the top 4.4 or 7.3 per cent [by citation impact] is clearly very high performance…but to claim that the amount of ‘world-leading’ research has doubled over that period lacks credibility and validity internationally.”

paul.jump@tesglobal.com

You've reached your article limit.

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Related articles

Related universities

POSTSCRIPT:

Print headline: REF triumph of life sciences ‘lacks credibility’

Have your say

Log in or register to post comments

Most Commented

Worried man wiping forehead
Two academics explain how to beat some of the typical anxieties associated with a doctoral degree

Felipe Fernández-Armesto takes issue with a claim that the EU has been playing the sovereignty card in Brexit negotiations

John McEnroe arguing with umpire. Tennis

Robert MacIntosh and Kevin O’Gorman explain how to negotiate your annual performance and development review

Man throwing axes

UCU attacks plans to cut 171 posts, but university denies Brexit 'the reason'

Cricket player and umpire exchanging bribe

The need to accommodate foreign students undermines domestic practices, says Lincoln Allison, spying parallels between UK universities and global sports bodies such as Fifa