A mantra for the next REF cycle? Keep calm and carry on

We must improve the system we already have, not make a switch to metrics, argues James Wilsdon

January 21, 2016
Miles Cole illustration (21 January 2016)
Source: Miles Cole

After a long winter of consultation, even the most battle-hardened of higher education policy wonks could be forgiven some weariness at the prospect of yet another review. But a fresh look at the research excellence framework by Lord Stern and colleagues is a welcome opportunity to settle an argument that has been festering for too long.

Assuming that we want to retain a dual-support funding system – and the government has given every indication that it does – how do we allocate the quality-related portion of that funding (now about £1.6 billion each year) in a way that is accountable, efficient and rewards high-quality, high-impact research?

Across the sector, the REF has become a condensation point for wider concern about the burden of audit, management and bureaucracy on academic life. An exaggerated debate has developed, in which the REF is frequently discussed in rather hysterical terms as “a bloated boondoggle”, a “Frankenstein monster” or “a Minotaur that must be appeased by bloody sacrifices”. It is supposedly responsible for a “blackmail culture”, a “fever” and a “toxic miasma” on campus.

It is no surprise that these messages have filtered up the system to ministers, and created a perception that the REF is a problem that needs fixing. So a cool, hard and evidence-informed look at the costs and benefits of the exercise – and a consideration of alternative approaches – is welcome. The Stern Review should also fill in the remaining pieces of the research policy jigsaw, after the higher education Green Paper and Nurse Review of the research councils left several questions unresolved.

As Jonathan Grant pointed out in Times Higher Education earlier this month, even at an estimated cost of £246 million in 2014, the REF is actually a considerably more efficient means of distributing research funding than the research council grants system. The discrepancy will only be exacerbated by the fact that the declining value of the science budget has pushed success rates in some research council funding schemes to below 10 per cent, with the result that nine in 10 applicants are investing large amounts of time and effort in unsuccessful proposals.

The thorough series of evaluations of REF 2014 carried out last year will provide a solid evidence base on which to build. In addition, my independent review of metrics, published as The Metric Tide in July 2015, took an in-depth look at the role that quantitative data could play.

REF sceptics often look to metrics as the answer. This is reflected in the Stern Review’s terms of reference, which call for “a simpler, lighter-touch method of research assessment, that more effectively uses data and metrics while retaining the benefits of peer review”.

But as we’re seeing in debates over metrics for teaching assessment, the seductive promise of big data doesn’t always hold up under scrutiny. My review group approached this issue with an entirely open mind, but after spending 15 months analysing available metrics, performing correlation analyses and consulting widely, we were forced to conclude that “no set of numbers, however broad, is likely to be able to capture the multifaceted and nuanced judgements on the quality of research outputs that the REF process currently provides”.

If all the REF is intended to do is provide an allocation mechanism for quality-related funding then it’s perfectly possible to devise an algorithmic solution. But the exercise, as it has evolved through successive cycles, does a lot more than this. And compared with the existing process, coverage and robustness across disciplines remains the biggest challenge for metrics. Whatever Elsevier and others with metric services to sell may say, there are serious gaps, especially across the arts, humanities and social sciences.

For instance, my review highlighted that there are a handful of arts-based higher education institutions for which journal articles were a minor component of their 2014 REF submission. And while the percentage of non-journal outputs may be much smaller for larger research-intensives (24 per cent at the University of Oxford; 15 per cent at University College London) their sheer size means a great deal of excellent research would not be counted if we switched to a purely citation-based approach.

Regarding the impact element of the REF, the problem is even greater. As both my review and a separate evaluation by King’s College London and Digital Science show, we simply don’t have reliable quantitative indicators that can capture the richness and diversity of impacts depicted in the near 7,000 REF case studies.

So while the Stern Review’s aspiration to reduce burden and incorporate quantitative data more intensively in research evaluation is entirely correct, metrics provide few easy wins. And, as Grant also pointed out, the financial savings from a switch to metrics are likely to be modest at best.

Instead of diving headlong into the metric tide, I continue to think that the focus for the next REF cycle should be on improving the robustness, coverage and interoperability of the datasets that we have, and applying them responsibly in the management of the research system. This need for a framework of “responsible metrics” for research was the headline message of The Metric Tide, and I hope Lord Stern and colleagues will consider it seriously.

James Wilsdon is professor of research policy and director of impact and engagement in the Faculty of Social Sciences at the University of Sheffield. An updated book edition of The Metric Tide, the report of his independent metrics review, is being published by Sage Publishing this week.

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

POSTSCRIPT:

Print headline: Nip and tuck

Reader's comments (1)

The problem is the 'we' with which this piece is peppered. Who are the 'we' ? The REF works for some to the detriment of others in the universities subject to or complicit in it. There is not a unity of interests generally in universities in England nor in relation to the REF. There is no 'we'.

Have your say

Log in or register to post comments

Featured Jobs

Professor-Keith Cameron Chair of Australian History UNIVERSITY COLLEGE DUBLIN (UCD)
Senior Procurement Officer UNIVERSITY OF THE WEST OF SCOTLAND
Clinician, Small Animal Emergency Services UNIVERSITY COLLEGE DUBLIN (UCD)
Director COVENTRY UNIVERSITY COLLEGE

Most Commented

question marks PhD study

Selecting the right doctorate is crucial for success. Robert MacIntosh and Kevin O'Gorman share top 10 tips on how to pick a PhD

India, UK, flag

Sir Keith Burnett reflects on what he learned about international students while in India with the UK prime minister

Pencil lying on open diary

Requesting a log of daily activity means that trust between the institution and the scholar has broken down, says Toby Miller

Construction workers erecting barriers

Directly linking non-EU recruitment to award levels in teaching assessment has also been under consideration, sources suggest

Microlight pilot flies with flock of cranes

Reports of UK-based researchers already thinking of moving overseas after Brexit vote