Blind faith in metrics is 'unfounded'

Warning about 'subjectivity' of data raises concerns about the REF's use of indices, writes Zoe Corbyn

June 26, 2008

The faith being placed in the use of statistics to provide accurate judgment of the quality of academic research is "unfounded", says a group of leading international statisticians.

A report produced by the International Mathematical Union (IMU) warns against the "misuse of statistics in assessing scientific research".

It says that faith in the accuracy, independence and efficacy of metrics on their own is misplaced. Numbers are not "inherently superior" to sound judgments and can be "even more subjective than peer review".

The report, Citation Statistics, looks at numerical measures commonly used to rank researchers, which it says are being increasingly used the world over as a "simple and objective" way to judge quality.

It examines the use of journal impact factors, which assess research based on the relative standing of the journal in which it is published, and citation counts, which measure output based on the number of times an academic's published work is cited by his or her peers.

Citation counts will be used, in combination with a form of peer review, in the UK's forthcoming research excellence framework, which replaces the research assessment exercise to determine the allocation of more than £1 billion of research funding to universities each year.

"While numbers appear to be 'objective', their objectivity can be illusory ... Because this subjectivity is less obvious for citations, those who use citation data are less likely to understand their limitations," the report concludes.

It adds that such indicators should be used only as part of a wider package that includes peer review and, possibly, other "esteem measures", such as conference invitations and editorial board memberships.

"They need to be supplemented," said John Ewing, the chair of the committee that produced the report and executive director of the American Mathematical Society. He added that people should be alert to the fact that various groups - such as the scientific disciplines set to benefit most from citations and bibliometrics analysts - had vested interests in pushing for citation-based statistics.

John Blake, head of mathematics at the University of Birmingham, described the report as a "wake-up call" for the UK.

"There is a general concern internationally that citations and indices are starting to have too great an impact in certain areas. The concern is that they might start to play too great a role in the UK," he said.

A spokesman for the Higher Education Funding Council for England said the report raised issues that it had already been considering in developing the REF.

Register to continue

Why register?

  • Registration is free and only takes a moment
  • Once registered, you can read 3 articles a month
  • Sign up for our newsletter
Please Login or Register to read this article.