Life sciences' REF triumph 'lacks credibility'

Bibliometric analysis concludes lower-quality thresholds in 2014 are more likely explanation for doubling of top-rated research

July 27, 2015
Tarsier has eyes that are larger than it's brain. Bohol island, Philippines
Source: Alamy
Stardust: 4* work doubled in six years

The implication of the research excellence framework results that the amount of world-leading research carried out in the UK has doubled in the past six years “lacks credibility”.

This is the claim of a team led by Jonathan Grant, director of the Policy Institute at King’s College London, which used bibliometrics to examine the apparently “remarkable improvement” in UK life science research that saw a doubling in the proportion of work graded 4* (“world-leading”) in the 2014 REF, compared with the 2008 research assessment exercise (rising from 11.8 to 23.9 per cent of the total).

The team found an increase of only 25 per cent in the proportion of submitted papers that were in the top 10 per cent of global papers published in the same field in the same year. Since 9 per cent fewer life science articles were submitted to the REF, the absolute number of papers in the top 10 per cent was just 10 per cent higher.

The results are recorded in a paper, “UK doubles its ‘world-leading’ research in life sciences and medicine in six years: testing the claim?”, published in Plos One on 23 July.

The number of life sciences articles rated 4* in the RAE was the same as the number of submitted papers that were in the top 4.4 per cent of global papers ranked by citation impact. That figure was 7.3 per cent for the REF. This, according to the paper, suggests that the main reason for grade inflation in the REF was that its panels used “a somewhat lower threshold of acceptance for a 4* level”.

By comparison, the difference between thresholds in the physical sciences was less marked. The number of such papers rated 4* in the RAE was the same as the number in the top 6.5 per cent globally, compared with 7.7 per cent in the REF.

Thresholds may have been lowered, the paper suggests, due to the increased reward in 2014 for 4* papers in the formula allocating quality-related research funding. Since the REF results “have huge implications for the relative standing of fields, the research funding of universities, and funding allocated within universities to different research groups” it is “understandable…the REF evaluations are likely to be influenced by these external factors”.

Metrics are, of course, controversial. But the paper argues that, especially at a large scale, they can “provide an external indicator of research quality”.

“For the advancement of science and health, evaluation of research quality requires consistency and validity [and] the discrepancy noted here calls for a closer examination of mass peer-review methods like the REF,” it adds.

Speaking to Times Higher Education, Professor Grant added that grade inflation “makes it harder to distinguish between different degrees of ‘excellence’ within the 4* category”.

This “potentially means the performance-related funding aspects are undermined and the incentive to grow true quality is diluted”, he said.

“Being in the top 4.4 or 7.3 per cent [by citation impact] is clearly very high performance…but to claim that the amount of ‘world-leading’ research has doubled over that period lacks credibility and validity internationally.”

paul.jump@tesglobal.com

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Related articles

Related universities

POSTSCRIPT:

Print headline: REF triumph of life sciences ‘lacks credibility’

Have your say

Log in or register to post comments

Featured Jobs

PhD Scholar in Medicine

University Of Queensland

Manager, Research Systems and Performance

Auckland University Of Technology

Lecturer in Aboriginal Allied Health

University Of South Australia

Lecturer, School of Nursing & Midwifery

Western Sydney University

College General Manager, SHE

La Trobe University
See all jobs

Most Viewed

Most Commented

Lady Margaret Hall, Oxford will host a homeopathy conference next month

Charity says Lady Margaret Hall, Oxford is ‘naive’ to hire out its premises for event

women leapfrog. Vintage

Robert MacIntosh and Kevin O’Gorman offer advice on climbing the career ladder

Woman pulling blind down over an eye
Liz Morrish reflects on why she chose to tackle the failings of the neoliberal academy from the outside
White cliffs of Dover

From Australia to Singapore, David Matthews and John Elmes weigh the pros and cons of likely destinations

Mitch Blunt illustration (23 March 2017)

Without more conservative perspectives in the academy, lawmakers will increasingly ignore and potentially defund social science, says Musa al-Gharbi