Researchers raise questions over validity of REF citations criteria

Two leading researchers claimed this week that the system that is to replace the research assessment exercise would systematically underestimate the impact of their research and could disadvantage their subjects.

The Higher Education Funding Council for England has confirmed that from 2009 research quality in sciences will be judged on the basis of the number of times academics' research is cited by their peers.

Two researchers in astronomy and physics at Durham University, backed by the head of research, have calculated their potential citation counts: they used the citation databases standard in their fields and compared them with counts produced using the Thomson Scientific Web of Science (WoS), the database Hefce intends to use for the research excellence framework (REF).

The count for Durham astronomer Carlos Frenk, the most highly cited astronomy and space scientist in the UK, was more than 5,000 short of the number cited in the Astrophysics Data System (a loss of about 18 per cent). For particle physicist Nigel Glover, nearly 1,600 citations counted by the standard particle physics citation database, Spires, did not show up in the WoS (a loss of about 37 per cent).

This was not the result of any flaws in the WoS, but because the WoS excludes citations in papers that are not yet formally published and citations in many conference proceedings - yet these can indicate a strong influence in the field. As a result, the researchers argue that their own subject databases, rather than the WoS, would be more appropriate to use in the REF.

James Stirling, Durham's pro vice-chancellor for research, said the examples "highlighted the danger of using a universal database to compare research in different areas".

Professor Frenk said: "By including conference proceedings and papers, the ADS is giving you a truer reflection of what you are trying to measure, which is impact."

Professor Glover added: "The design of the WoS fails to capture our culture in which pre-published papers are available to other researchers."

Jim Pringle, vice-president of development at Thomson Scientific, said the WoS "captures the citations generated by the research community as it engages in formal scholarly communication via peer-reviewed journals". This consistency allows "apples-to-apples" comparisons fundamental to sound research metrics. He added that when communications took place in forums other than peer-reviewed journal literature "other resources may provide complementary data".

In the REF proposals, currently out for consultation with a closing date of 14 February, Hefce recognises that there may be "limitations" to WoS coverage for some subjects, but it does not identify either particle physics or astronomy as of particular concern. However, Rama Thirunamachandran, director of research at Hefce, said the organisation was keeping an open mind.

"We have commissioned a study - reporting in spring - to look at the potential to supplement the WoS with other data sources. It will look not just at how far other sources can extend coverage but also at important issues of data quality".

zoe.corbyn@tsleducation.com