Pros and cons of the metrics system

May 16, 2013

The main problem with the research excellence framework is the huge amount of time and money that it consumes, distracting academics from scholarly activities and diverting funds to employ squads of administrators to manage it (“For richer, for poorer”, 9 May).

In a recent blog post, I reported an analysis I did showing that you could pretty accurately predict the research assessment exercise’s psychology results by taking an H-index (indexes that attempt to measure the productivity and impact of academics’ published work) based on departmental addresses.

In the comments on the post, there is further evidence that this is also true for physics. Computing the H-index takes an individual two to three hours, as opposed to the two to three years spent by legions of people working on the REF.

At present, we don’t know whether such differences as exist between H- index ratings and RAE panel ratings mean that the latter were better. For both psychology and physics, once you had taken the H-index into account, additional variance in funding levels could be explained by whether there were departmental representatives on the assessment panels.

This suggests that the additional value given by having expert assessment incorporated in such evaluations may just be adding subjective bias, which does not necessarily indicate any malpractice but could reflect the advantage that panel members have of intimate knowledge of how the panels work.

Most of us don’t like metrics, and I accept that they may not work in the humanities, but I would suggest that if we are not going to use a metric like this, we need to do an analysis to justify the additional cost incurred by any alternative procedure. If the additional cost is very high, then we might decide that it is preferable to use a metric-based system, warts and all, and to divide the money saved between all institutions.

Dorothy Bishop
Via timeshighereducation.co.uk

 

One of the problems with the use of citations as a measure of research quality is that the method assumes that the higher the number, the greater the quality. Ignoring the possibility of “tit for tat” reciprocity between mates, what if your article is cited and immediately preceded by “for a total misunderstanding of even these basics, see Mead…”?

In addition, citations don’t work as a measure of anything where the chance of others quoting your work is low: if I were ploughing a relatively lonely research furrow, I’d prefer to take my chances with a subpanel of the great and the good. The fact that no one else is relying on my work because no one else is interested in it (or has even heard of it) says nothing about whether it is good, bad or indifferent. Like impact, citations therefore have a tendency to skew personal research interests into institutional research agendas, favouring more of the greatly populated same, not those who are pushing boundaries and exploring for its own sake.

David Mead
Professor of public law and UK human rights
University of Essex

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Featured Jobs

Head Coach of Women's Basketball

California University Of Technology

Employee Relations Consultant

Macquarie University - Sydney Australia

Administrative Assistant

University Of Wollongong

Lecturer - Faculty Office

Monash University

Senior People & Culture Advisor

Swinburne University Of Technology
See all jobs

Most Commented

Laurel and Hardy sawing a plank of wood

Working with other academics can be tricky so follow some key rules, say Kevin O'Gorman and Robert MacIntosh

Lady Margaret Hall, Oxford will host a homeopathy conference next month

Charity says Lady Margaret Hall, Oxford is ‘naive’ to hire out its premises for event

women leapfrog. Vintage

Robert MacIntosh and Kevin O’Gorman offer advice on climbing the career ladder

Woman pulling blind down over an eye
Liz Morrish reflects on why she chose to tackle the failings of the neoliberal academy from the outside
White cliffs of Dover

From Australia to Singapore, David Matthews and John Elmes weigh the pros and cons of likely destinations