Careers at risk after case studies ‘game playing’, REF study suggests

Submission patterns reveal how institutions responded to rules

January 22, 2015

Click here to view full graph


A significant number of academics’ careers could be tarnished by exclusion from the research excellence framework merely on the basis that their university could not find enough good impact case studies, an analysis indicates.

The number of case studies universities were required to submit to the 2014 REF was determined by the number of full-time equivalent (FTE) staff they submitted. Two case studies were required for submissions containing up to 14.99 staff, three from those between 15 and 24.99 staff, and so on.

Graeme Rosenberg, REF manager for the UK funding bodies, confirmed to Times Higher Education that his intention had been that decisions about the number of staff submitted would determine the number of case studies to submit, rather than vice versa.

However, an analysis of the REF results by Tim Horne, head of the Research Excellence Unit at Coventry University, reveals that the number of submissions containing staff numbers just below the threshold for an extra case study was far higher than would be expected statistically.

Dr Horne rounded down to whole numbers the number of staff in each of the 1,911 submissions to the REF, and plotted how frequently each number appeared. He found a series of sharp spikes in numbers ending with a four (view full graph).

Dr Horne said that his analysis confirmed what “everybody knew would happen anyway: that you would get institutions determining the number of people going into a submission by the number of case studies they had that they thought were good”.

He said the same pattern was evident for all mission groups and main panels. Examination of individual institutions’ submission patterns risked not being statistically significant. But he noted that of Durham University’s 23 submissions, 18 were within one FTE of a case study boundary. The institution was 19th by grade point average on impact, and 20th overall, submitting 79 per cent of its eligible staff.

Universities generally scored highly for impact: the sector’s overall grade point average was 3.24, compared with 2.90 for outputs. But Patrick Dunleavy, professor of political science and public policy at the London School of Economics, said it was “very disturbing” that universities appeared to have been “gaming the system” by not submitting individuals whose outputs otherwise merited submission. Universities typically insist that non-submission to the REF will not, in itself, have negative career consequences. But many academics remain sceptical of such claims: “Individual careers may well have been sacrificed for institutional convenience,” Professor Dunleavy said.

Claire Warwick, pro vice-chancellor for research at Durham, said that her institution had aimed to “make the best submission possible” within the rules.

“Decisions about whether [the] rules might be changed in future are a matter for the funding bodies to consider. We would suggest that if they wish to measure research intensiveness in a future REF, they might require universities to enter all academic staff,” she said.

Professor Dunleavy also advocated universal submission but preferred a metrics-based approach to avoid “raising the already huge workload” and to end the “current farce of assessment by eyeballing”.

Dr Rosenberg said that he was “aware that the case study requirements had an effect on submission size. We will need to look at the implications and consider this issue for the future.”

paul.jump@tesglobal.com

Times Higher Education free 30-day trial

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 10 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Featured Jobs

Business Development Officer YORK ST JOHN UNIVERSITY
Lecturer in Linguistics YORK ST JOHN UNIVERSITY
Associate Professor/Professor EDINBURGH NAPIER UNIVERSITY

Most Commented

Brexit, EU referendum

Joanna Williams voted Leave, and has been left disappointed by the academy’s reaction to the EU referendum result

Elly Walton illustration (7 July 2016)

Researchers in the liberal arts seem to have made it their mission to communicate in the most obscure fashion, says Zachary Foster

Daniel Mitchell illustration (14 July 2016)

Frank Furedi says the mournful mood on campus and the disparagement and silencing of Leave supporters betray an isolated scholarly class

Michael Parkin illustration (7 July 2016)

Rising immigration-related costs and lack of employer support send an unwelcoming message to international staff, says Jason Danely

Huge eyes peering through door windows

Some scholars claim plans are more about improving staff visibility to students than brightening corridors