Hefce backtracks to test post-RAE plans

The Higher Education Funding Council for England has returned almost to square one in the detailed design of the new system to replace the research assessment exercise.

In November, Hefce set out details of how the research excellence framework would assess research quality in the sciences. Quality would be judged using metrics, such as the number of times academics' published work was cited by peers.

Under the plans, all published papers from selected academic staff would be included, but self-citations - where an academic has cited his or her own work - would be excluded. Hefce also said that "initially at least" the Thomson Scientific Web of Science (WoS) would be used as the primary source of citation data.

Now, however, Hefce has said that these original proposals will be reviewed in a planned "pilot exercise". The backtrack follows an announcement by Hefce last month that the REF implementation timetable is to be extended by a year, and plans to split science from non-science subjects will be abandoned in favour of a "continuum" approach.

The further revisions were detailed last week by Graeme Rosenberg, Hefce's REF pilot manager, at a conference, "Beyond the RAE 2008: Bibliometrics, League Tables and the REF".

Mr Rosenberg said that while a recent consultation showed there was overall agreement to use citations per paper (adjusted for the field) as a quality indicator, and that the results should take the form of a "quality profile", other key issues had shown "less agreement".

On the system's scope, he said: "Are we looking at all staff or all papers in relevant disciplines? Or selected staff or selected papers? ... We don't have an answer to that yet. It will be something we test in the pilot phase."

Mr Rosenberg said the pilot would also consider whether published papers should be credited to the institution or should follow the researcher from institution to institution; how Hefce collected data on staff and their papers; and what role universities should have in managing their data.

Also to be decided will be how citations in fields are "normalised", and how multi-authorship and self-citation are handled.

"We had proposed that self-citation should be excluded. There was quite a lot of concern raised about that, and we will look into that again in the pilot," Mr Rosenberg said.

He said Hefce "had not taken any firm decision" about whether WoS or its competitor Scopus would be used to provide the citation database and both could be used in the pilot.

Jonathan Adams, director of the data analysis firm Evidence Ltd, expressed surprise at the WoS decision. "The question emerges as to what extent universities have already made investments based on the earlier announcement," he said.

Fifty-three higher education institutions have expressed an interest in participating in the pilot.

zoe.corbyn@tsleducation.com

COUNTDOWN TO THE REF

• May 2008: Survey of potential pilot institutions
• Early June 2008: Pilot institutions selected and informed
• August to October 2008: Data collection
• November 2008 to early 2009: Data analysis
• Spring 2009: Pilot consultation.