Metrics marred by doubt at first cite

September 21, 2007

When researcher Richard Hull searched online for citations of an article about German and American industrial relations, he did not expect to find links to Cell and Tissue Research and The Journal of Neuroscience , writes Louise Radnofsky.

The Web of Science online academic database wrongly assured him that the paper he was looking for was cited in articles from nine different biology journals.

The incident has raised more serious questions about the growing use of metrics - measurements of indicators such as the number of times published research has been cited - to judge the quality of academic work.

The research assessment exercise, which the Higher Education Funding Council for England uses to determine how to allocate billions of pounds of funding among universities, will be largely based on metrics after 2008.

"The whole metrics system is predicated on the reliability of the citations index," said Dr Hull, a senior lecturer in management at Newcastle University Business School. He said his finding "makes current government policy on research assessment look a bit dodgy".

He was "gobsmacked" by the error. "I'd never seen a mistake. I had absolutely taken it for granted that this was God's honest truth. If they say the article's been cited 174 times, then they're right," he said.

Dr Hull reported the error to the website. It was agreed that there had probably been a "serious problem with the database", and Thomson Scientific, which maintains the database, said it had been corrected.

Richard Hill, of Thomson Scientific's technical support team, said that errors could arise from the way a citation had been written. But he added that he had "no doubt" that errors were sometimes introduced when the articles were entered into the database. A Thomson Scientific spokeswoman said article linking errors did not affect journal-level metrics.

A Hefce spokesman said it was taking expert advice on the use of citations as it prepared for a new research assessment system.

Register to continue

Why register?

  • Registration is free and only takes a moment
  • Once registered, you can read 3 articles a month
  • Sign up for our newsletter
Register
Please Login or Register to read this article.

Sponsored