The metrics of metrics 1

February 28, 2008

Critics of the Cranfield study of correlations between 2001 research assessment exercise scores and citation counts (Letters, 21 February) ignore two important points. The first is that a metric based entirely on research outputs makes more sense than a measure that uses research inputs if the aim is to measure the quality rather than the cost of research. The second is that the metric used (average number of citations per article submitted to the 2001 RAE) measures in a very transparent way the impact of the published work that was used by RAE panels to assess research quality.

It was therefore fascinating to see that this objective, metric-based assessment led to such different results from the rather less transparent system preferred by the Higher Education Funding Council for England at that time, that is, peer review combined with measures of research inputs. Less than a quarter of subjects showed a good correlation with 2001 RAE ratings and even those with good correlations had significant anomalies.

I should declare an interest here. In chemistry, Huddersfield and several other post-Robbins universities, including Warwick and York, performed substantially better on the Cranfield bibliometric measure than predicted by their RAE scores, even though there was a good correlation overall.

Other metrics, such as total number of citations per department or research grants awarded, tell us more about size of unit and how expensive the research is than about research quality. Research inputs do help inform assessment of funding needs but, at least in science, technology, engineering and medicine subjects where journal publication is the norm, published outputs should provide the primary means of measuring quality.

Rob Smith
Professor and dean of applied sciences
University of Huddersfield

You've reached your article limit.

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments

Have your say

Log in or register to post comments

Most Commented

James Fryer illustration (27 July 2017)

It is not Luddism to be cautious about destroying an academic publishing industry that has served us well, says Marilyn Deegan

Jeffrey Beall, associate professor and librarian at the University of Colorado Denver

Creator of controversial predatory journals blacklist says some peers are failing to warn of dangers of disreputable publishers

Hand squeezing stress ball
Working 55 hours per week, the loss of research periods, slashed pensions, increased bureaucracy, tiny budgets and declining standards have finally forced Michael Edwards out
Kayaker and jet skiiers

Nazima Kadir’s social circle reveals a range of alternative careers for would-be scholars, and often with better rewards than academia

hole in ground

‘Drastic action’ required to fix multibillion-pound shortfall in Universities Superannuation Scheme, expert warns