Careers at risk after case studies ‘game playing’, REF study suggests

Submission patterns reveal how institutions responded to rules

January 22, 2015

Click here to view full graph


A significant number of academics’ careers could be tarnished by exclusion from the research excellence framework merely on the basis that their university could not find enough good impact case studies, an analysis indicates.

The number of case studies universities were required to submit to the 2014 REF was determined by the number of full-time equivalent (FTE) staff they submitted. Two case studies were required for submissions containing up to 14.99 staff, three from those between 15 and 24.99 staff, and so on.

Graeme Rosenberg, REF manager for the UK funding bodies, confirmed to Times Higher Education that his intention had been that decisions about the number of staff submitted would determine the number of case studies to submit, rather than vice versa.

However, an analysis of the REF results by Tim Horne, head of the Research Excellence Unit at Coventry University, reveals that the number of submissions containing staff numbers just below the threshold for an extra case study was far higher than would be expected statistically.

Dr Horne rounded down to whole numbers the number of staff in each of the 1,911 submissions to the REF, and plotted how frequently each number appeared. He found a series of sharp spikes in numbers ending with a four (view full graph).

Dr Horne said that his analysis confirmed what “everybody knew would happen anyway: that you would get institutions determining the number of people going into a submission by the number of case studies they had that they thought were good”.

He said the same pattern was evident for all mission groups and main panels. Examination of individual institutions’ submission patterns risked not being statistically significant. But he noted that of Durham University’s 23 submissions, 18 were within one FTE of a case study boundary. The institution was 19th by grade point average on impact, and 20th overall, submitting 79 per cent of its eligible staff.

Universities generally scored highly for impact: the sector’s overall grade point average was 3.24, compared with 2.90 for outputs. But Patrick Dunleavy, professor of political science and public policy at the London School of Economics, said it was “very disturbing” that universities appeared to have been “gaming the system” by not submitting individuals whose outputs otherwise merited submission. Universities typically insist that non-submission to the REF will not, in itself, have negative career consequences. But many academics remain sceptical of such claims: “Individual careers may well have been sacrificed for institutional convenience,” Professor Dunleavy said.

Claire Warwick, pro vice-chancellor for research at Durham, said that her institution had aimed to “make the best submission possible” within the rules.

“Decisions about whether [the] rules might be changed in future are a matter for the funding bodies to consider. We would suggest that if they wish to measure research intensiveness in a future REF, they might require universities to enter all academic staff,” she said.

Professor Dunleavy also advocated universal submission but preferred a metrics-based approach to avoid “raising the already huge workload” and to end the “current farce of assessment by eyeballing”.

Dr Rosenberg said that he was “aware that the case study requirements had an effect on submission size. We will need to look at the implications and consider this issue for the future.”

paul.jump@tesglobal.com

Times Higher Education free 30-day trial

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Most Commented

Lady Margaret Hall, Oxford will host a homeopathy conference next month

Charity says Lady Margaret Hall, Oxford is ‘naive’ to hire out its premises for event

women leapfrog. Vintage

Robert MacIntosh and Kevin O’Gorman offer advice on climbing the career ladder

Woman pulling blind down over an eye
Liz Morrish reflects on why she chose to tackle the failings of the neoliberal academy from the outside
White cliffs of Dover

From Australia to Singapore, David Matthews and John Elmes weigh the pros and cons of likely destinations

Steve Keen, Kingston University

Steve Keen laments loss of ‘time and freedom’ for universities’ ‘original thinkers’