Skip the REF and count the notes?

Citations could replace peer review in sciences, study concludes

July 18, 2013

Source: Getty

Op. cit. effect: departmental citation counts correlate with RAE ratings

The research excellence framework could be replaced by citation analysis, but only in the hard sciences and only if results are not used to rank quality, a study has concluded.

Four physicists and mathematicians compared the results of the 2008 research assessment exercise - the REF’s forerunner - with citation data from Thomson Reuters.

In a paper published on the arXiv pre-print server, “Comparison of a citation-based indicator and peer review for absolute and specific measures of research-group excellence”, the authors find a high correlation between the total citation count for papers submitted by a department to the RAE and that department’s quality rating determined by the assessment panels.

The correlation is particularly high for larger groups in the hard sciences.

The REF is intended primarily to inform the distribution of quality- related research funding based on an aggregation of peer-review scores at departmental and institutional level. For this reason, the paper says that “the use of citation-based indicators may offer a much cheaper, and less intrusive alternative to [peer review] for large research groups in the hard sciences”. But it cautions that citation analysis would be “far less reliable for the social sciences and humanities”.

One of the paper’s authors, Ralph Kenna, deputy director of Coventry University’s Applied Mathematics Research Centre, told Times Higher Education that in practice it would be “unacceptable” to replace peer review with citation counts in some subjects and not others because departments might try to choose which panel to submit to based on whether they expected to do better via peer review or citation analysis.

“It would stymie curiosity-driven research and further distort the academic landscape,” he said.

The paper also warns against using citations as a measure of departmental or institutional quality. This is because quality should be a per-head measure, but “almost paradoxically” citations-per-head correlate only weakly with RAE quality scores.

The formal inclusion of citation analysis in the 2014 REF was rejected after a feasibility study on the grounds that citations were “not sufficiently robust” to replace peer review - including in the sciences. According to Graeme Rosenberg, REF manager for the Higher Education Funding Council for England, worries included anomalies such as negative citations, the possibility of self-citation and “citation clubs” and the time lag between publication and the accumulation of citations.

However, the issue is likely to be revisited ahead of the next REF, which is expected in 2020.

paul.jump@tsleducation.com

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Featured Jobs

Professor-Keith Cameron Chair of Australian History UNIVERSITY COLLEGE DUBLIN (UCD)
Senior Procurement Officer UNIVERSITY OF THE WEST OF SCOTLAND
Clinician, Small Animal Emergency Services UNIVERSITY COLLEGE DUBLIN (UCD)
Director COVENTRY UNIVERSITY COLLEGE

Most Commented

question marks PhD study

Selecting the right doctorate is crucial for success. Robert MacIntosh and Kevin O'Gorman share top 10 tips on how to pick a PhD

India, UK, flag

Sir Keith Burnett reflects on what he learned about international students while in India with the UK prime minister

Pencil lying on open diary

Requesting a log of daily activity means that trust between the institution and the scholar has broken down, says Toby Miller

Construction workers erecting barriers

Directly linking non-EU recruitment to award levels in teaching assessment has also been under consideration, sources suggest

Application for graduate job
Universities producing the most employable graduates have been ranked by companies around the world in the Global University Employability Ranking 2016