Hefce reveals full details of citation replacement for research assessment exercise. Zoe Corbyn reports. Academics will have to have their research papers cited by their peers at least three times more than the average rate if they are to meet the new standard for "international excellence" in research.
The standard will be part of the new Research Excellence Framework that will determine the allocation of about £1.4 billion of research funding to English universities, starting from 2010.
As The Times Higher reported earlier this month, English funding chiefs will, after next year's research assessment exercise, move to a new system that will measure research quality in science subjects according to the number of citations selected academics' published papers receive.
According to a 24-page consultation document published this week that sets out the detail of the new system, this assessment will lead to a "quality profile" for each of six large subject areas in each university.
The data will be combined with research income and postgraduate student numbers to determine the size of block research grants.
Arts and humanities subjects will continue to be judged on the basis of peer review.
But in the document, the Higher Education Funding Council for England said that it would "continue to explore the potential for bibliometric indicators to play a greater role in quality assessment in these subjects".
"Our proposals for citation analysis rest upon the principle that, when viewed in aggregate, the research outputs in any discipline that are most highly cited by other researchers will generally be those (with) the greatest intellectual influence," says the consultation document.
Under Hefce's plans, universities would select which of their staff to include in the process by early 2009.
The number of times selected academics' papers are cited by others - in a period going back between five and ten years that is yet to be decided - would then be determined, "normalised" to account for different citation levels between fields and then added together to determine a profile for each broad science subject area. All papers from the selected academics would be included.
"While the average number of citations per paper varies widely between disciplines, it will be reasonable to measure relative levels of excellence on a common basis once normalisation by field has been performed," Hefce says. "It is reasonable to assume that material receiving citations at three times the field average rate is of a broadly equivalent level of international excellence across all disciplines."
The primary source of citation data will be the Thomson Scientific Web of Science, although Hefce notes there are limitations inherent in this method that will require further investigation.
Hefce proposes drawing up quality profiles for science-based disciplines in six broadly defined subject groups: clinical medicine, health services, subjects allied to health, biological sciences, physical sciences and engineering and computer science. This is fewer than the RAE units of assessment, and groups will be broader.
Citation rates would be "normalised" to account for differences between subject areas, using about 170 separate fields within the Web of Science database.
Rama Thirunamachandran, director of research and knowledge transfer at Hefce, said the new system would shift the current focus away from departments and individuals and reduce game-playing.
"Citations per paper is not a target one can easily influence on one's own. Peers - hundreds of them globally - judge what you are doing. The best one can do is to focus on producing work that is of excellent quality, cutting edge, interesting and innovative, and then let peers judge it by whether they cite it."
But the Hefce consultation also acknowledges the concerns that have emerged in the sector about the potential impact on researchers' behaviour that the new system could have, such as the formation of "citation clubs" in which researchers do deals to cite each others' papers.
"We are advised that the scope for manipulation is limited in practice," the Hefce consultation document says.
"We have expert advice that the scope for influencing citation impact through (citation clubs) is not significant within our preferred approach."
It adds that monitoring potential changes in citation behaviour will be possible against the current baseline, and it will consider introducing a mechanism for the reporting of "suspicious citation behaviour". It also proposes excluding from the data any cases in which an academic has cited his or her own work.
Along with the consultation, Hefce has also issued two reports on the new system. The first is a scoping study on the use of bibliometrics by a team at the University of Leiden that Hefce has heavily drawn on in formulating the consultation.
The second is a study by the UK firm Evidence Ltd that analyses how interdisciplinary research is likely to be affected by the new system. It found "no evidence that citation rates are systematically lower for interdisciplinary research".
The consultation also looks at the role of expert panels that oversee the system and asks for volunteer universities to pilot the Research Excellence Framework alongside the 2008 RAE.
The Hefce consultation closes on February 14. More details at www.hefce.ac.uk
MAIN POINTS
- After the 2008 RAE, "metrics" will be introduced gradually between 2010 and 2014;
- For science disciplines, quality will be measured by the number of times that research papers are cited by other researchers;
- Institutions will select which staff they wish to include;
- "Suspicious" citation behaviour, designed to manipulate results, will be monitored;
- In arts and humanities, quality will be judged by a "light-touch" peer review, which will take place in 2013;
- For arts and humanities, Hefce will "continue to explore the potential for bibliometric indicators to play a greater role in quality assessment".