Careers at risk after case studies ‘game playing’, REF study suggests

Submission patterns reveal how institutions responded to rules

January 22, 2015

Click here to view full graph

A significant number of academics’ careers could be tarnished by exclusion from the research excellence framework merely on the basis that their university could not find enough good impact case studies, an analysis indicates.

The number of case studies universities were required to submit to the 2014 REF was determined by the number of full-time equivalent (FTE) staff they submitted. Two case studies were required for submissions containing up to 14.99 staff, three from those between 15 and 24.99 staff, and so on.

Graeme Rosenberg, REF manager for the UK funding bodies, confirmed to Times Higher Education that his intention had been that decisions about the number of staff submitted would determine the number of case studies to submit, rather than vice versa.

However, an analysis of the REF results by Tim Horne, head of the Research Excellence Unit at Coventry University, reveals that the number of submissions containing staff numbers just below the threshold for an extra case study was far higher than would be expected statistically.

Dr Horne rounded down to whole numbers the number of staff in each of the 1,911 submissions to the REF, and plotted how frequently each number appeared. He found a series of sharp spikes in numbers ending with a four (view full graph).

Dr Horne said that his analysis confirmed what “everybody knew would happen anyway: that you would get institutions determining the number of people going into a submission by the number of case studies they had that they thought were good”.

He said the same pattern was evident for all mission groups and main panels. Examination of individual institutions’ submission patterns risked not being statistically significant. But he noted that of Durham University’s 23 submissions, 18 were within one FTE of a case study boundary. The institution was 19th by grade point average on impact, and 20th overall, submitting 79 per cent of its eligible staff.

Universities generally scored highly for impact: the sector’s overall grade point average was 3.24, compared with 2.90 for outputs. But Patrick Dunleavy, professor of political science and public policy at the London School of Economics, said it was “very disturbing” that universities appeared to have been “gaming the system” by not submitting individuals whose outputs otherwise merited submission. Universities typically insist that non-submission to the REF will not, in itself, have negative career consequences. But many academics remain sceptical of such claims: “Individual careers may well have been sacrificed for institutional convenience,” Professor Dunleavy said.

Claire Warwick, pro vice-chancellor for research at Durham, said that her institution had aimed to “make the best submission possible” within the rules.

“Decisions about whether [the] rules might be changed in future are a matter for the funding bodies to consider. We would suggest that if they wish to measure research intensiveness in a future REF, they might require universities to enter all academic staff,” she said.

Professor Dunleavy also advocated universal submission but preferred a metrics-based approach to avoid “raising the already huge workload” and to end the “current farce of assessment by eyeballing”.

Dr Rosenberg said that he was “aware that the case study requirements had an effect on submission size. We will need to look at the implications and consider this issue for the future.”

Times Higher Education free 30-day trial

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments

Have your say

Log in or register to post comments

Most Commented

question marks PhD study

Selecting the right doctorate is crucial for success. Robert MacIntosh and Kevin O'Gorman share top 10 tips on how to pick a PhD

Pencil lying on open diary

Requesting a log of daily activity means that trust between the institution and the scholar has broken down, says Toby Miller

India, UK, flag

Sir Keith Burnett reflects on what he learned about international students while in India with the UK prime minister

Application for graduate job
Universities producing the most employable graduates have been ranked by companies around the world in the Global University Employability Ranking 2016
Construction workers erecting barriers

Directly linking non-EU recruitment to award levels in teaching assessment has also been under consideration, sources suggest