Universities need a REF that rewards potential

Lord Stern should devise a research excellence framework that looks beyond past performance, says Giosuè Baggio

May 12, 2016
Rows of multi-coloured AA batteries
Source: iStock

The most radical idea being suggested to Lord Stern regarding his review of the research excellence framework appears to be that institutions should be required to enter all their academics. The suggestion, endorsed by the University of Cambridge, is intended to prevent the “gaming” around submission numbers that many consider to be the REF’s biggest flaw.

But, in my view, the review needs to address a bigger problem. The REF, in common with other global systems of academic evaluation, rewards past performance on the assumption that it predicts future success. This approach is so deeply rooted in academic culture that many believe no alternative is possible. However, rewarding performance is problematic, regardless of whether evaluations emphasise quantity, quality or some form of academic impact, such as citations.

An example of a country that emphasises quantity is Norway. Total resources for research in the country increased between 2004 and 2012, with allocation partly based on performance criteria. Publication output has duly grown – but quality and impact have stagnated. Meanwhile, if the concerns about gaming in the REF are justified, the UK’s widely trumpeted quality improvements since it was introduced may be equally illusory.

In short, the evidence indicates that rewarding performance does not necessarily lead to better science: it merely provides institutions with incentives to artificially inflate their metrics. We need a new model of research evaluation that rewards potential instead. Consider three scenarios, in which A and B are two research units competing for funding:

(1) A and B currently produce research of similar quality. A produces relatively little, but its quality is improving every year. B produces significantly more than A, but quality is decreasing.

(2) B scores marginally better than A (on whatever output measure), but uses significantly more resources than A.

(3) A and B perform similarly on all measures on average, but A’s performance has improved over the assessment period while B’s has worsened.

The current system would reward B’s declining performance in (1), B’s inefficient use of resources in (2), and would allocate both units the same amount in (3). This is because allocations do not trace how quantity, quality, impact and expenditure change over time. Doing this would give a much clearer picture of likely future performance, and would therefore be a much more sensible mechanism by which to distribute future research funding. In all three cases above, A’s potential for further growth would be rewarded.

By divesting from declining units, the new system would allocate a larger share of resources to those with the potential to grow and produce high-quality research, thus promoting the development of novel research programmes. The new system might decide not to fund units that have reached a “productivity ceiling” if it was clear that further investments would yield, at best, diminishing returns. Units that continually produce at “ceiling levels” might still be worth supporting if their results were qualitatively superior to those of their competitors, but not if quality or quantity were decreasing significantly.

The new system would not be immune from gaming, but it would be less prone to it. Manufacturing the signs of potential is intrinsically more difficult than inflating output. Would institutions withhold high-quality publications until towards the end of an assessment cycle, to signal productivity growth? There would be strong deterrents to doing that, especially in the sciences: scholars would never accept the risk of being scooped by others. Citation impact would not only be delayed by that but in effect lost, because most citations would go to the first publication to appear. Therefore, it would be a self-defeating strategy for institutions too.

It is very hard to say what the effect of such a radical new system would be on the existing distribution of research funding. For this reason, the current winners would surely lobby hard against it. But for any country truly serious about driving ongoing improvements in research regardless of the age and prestige of the institution in which it is carried out, my idea may just be worth considering.

Giosuè Baggio is associate professor in the department of language and literature at the Norwegian University of Science and Technology in Trondheim.

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Related universities

POSTSCRIPT:

Print headline: Realising REF’s potential

Have your say

Log in or register to post comments

Most Commented

question marks PhD study

Selecting the right doctorate is crucial for success. Robert MacIntosh and Kevin O'Gorman share top 10 tips on how to pick a PhD

Pencil lying on open diary

Requesting a log of daily activity means that trust between the institution and the scholar has broken down, says Toby Miller

India, UK, flag

Sir Keith Burnett reflects on what he learned about international students while in India with the UK prime minister

Application for graduate job
Universities producing the most employable graduates have been ranked by companies around the world in the Global University Employability Ranking 2016
Construction workers erecting barriers

Directly linking non-EU recruitment to award levels in teaching assessment has also been under consideration, sources suggest