Life sciences' REF triumph 'lacks credibility'

Bibliometric analysis concludes lower-quality thresholds in 2014 are more likely explanation for doubling of top-rated research

July 27, 2015
Tarsier has eyes that are larger than it's brain. Bohol island, Philippines
Source: Alamy
Stardust: 4* work doubled in six years

The implication of the research excellence framework results that the amount of world-leading research carried out in the UK has doubled in the past six years “lacks credibility”.

This is the claim of a team led by Jonathan Grant, director of the Policy Institute at King’s College London, which used bibliometrics to examine the apparently “remarkable improvement” in UK life science research that saw a doubling in the proportion of work graded 4* (“world-leading”) in the 2014 REF, compared with the 2008 research assessment exercise (rising from 11.8 to 23.9 per cent of the total).

The team found an increase of only 25 per cent in the proportion of submitted papers that were in the top 10 per cent of global papers published in the same field in the same year. Since 9 per cent fewer life science articles were submitted to the REF, the absolute number of papers in the top 10 per cent was just 10 per cent higher.

The results are recorded in a paper, “UK doubles its ‘world-leading’ research in life sciences and medicine in six years: testing the claim?”, published in Plos One on 23 July.

The number of life sciences articles rated 4* in the RAE was the same as the number of submitted papers that were in the top 4.4 per cent of global papers ranked by citation impact. That figure was 7.3 per cent for the REF. This, according to the paper, suggests that the main reason for grade inflation in the REF was that its panels used “a somewhat lower threshold of acceptance for a 4* level”.

By comparison, the difference between thresholds in the physical sciences was less marked. The number of such papers rated 4* in the RAE was the same as the number in the top 6.5 per cent globally, compared with 7.7 per cent in the REF.

Thresholds may have been lowered, the paper suggests, due to the increased reward in 2014 for 4* papers in the formula allocating quality-related research funding. Since the REF results “have huge implications for the relative standing of fields, the research funding of universities, and funding allocated within universities to different research groups” it is “understandable…the REF evaluations are likely to be influenced by these external factors”.

Metrics are, of course, controversial. But the paper argues that, especially at a large scale, they can “provide an external indicator of research quality”.

“For the advancement of science and health, evaluation of research quality requires consistency and validity [and] the discrepancy noted here calls for a closer examination of mass peer-review methods like the REF,” it adds.

Speaking to Times Higher Education, Professor Grant added that grade inflation “makes it harder to distinguish between different degrees of ‘excellence’ within the 4* category”.

This “potentially means the performance-related funding aspects are undermined and the incentive to grow true quality is diluted”, he said.

“Being in the top 4.4 or 7.3 per cent [by citation impact] is clearly very high performance…but to claim that the amount of ‘world-leading’ research has doubled over that period lacks credibility and validity internationally.”

paul.jump@tesglobal.com

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Related articles

Related universities

POSTSCRIPT:

Print headline: REF triumph of life sciences ‘lacks credibility’

Have your say

Log in or register to post comments

Featured Jobs

Most Commented

question marks PhD study

Selecting the right doctorate is crucial for success. Robert MacIntosh and Kevin O'Gorman share top 10 tips on how to pick a PhD

India, UK, flag

Sir Keith Burnett reflects on what he learned about international students while in India with the UK prime minister

Pencil lying on open diary

Requesting a log of daily activity means that trust between the institution and the scholar has broken down, says Toby Miller

Application for graduate job
Universities producing the most employable graduates have been ranked by companies around the world in the Global University Employability Ranking 2016
Retired academics calculating moves while playing bowls

Lincoln Allison, Eric Thomas and Richard Larschan reflect on the ‘next phase’ of the scholarly life