Nervous Hefce 'edging out' of REF citations

Key departure from RAE may go as process is delayed to assess impact plans. Zoë Corbyn reports

April 1, 2010

The use of citations to judge research quality could be dropped entirely from the research excellence framework, potentially ditching one of the key differences with its predecessor, the research assessment exercise.

A document published last week by funding chiefs, Initial Decisions on the REF, says that citations - the number of times academics' work is cited by their peers - may not be used in the forthcoming process to inform expert review.

The news comes as David Sweeney, director of research at the Higher Education Funding Council for England, signalled that the REF was likely to be delayed by at least another year to allow a measure of the economic and social impact of research to be included.

The proposed use of citation data, along with the inclusion of an impact measure, are the two major characteristics that differentiate the REF from the RAE.

The system will be used to distribute about £1.5 billion in research funding in England each year.

An earlier round of proposals on the RAE's replacement suggested the use of a "basket of metrics", including citations, to replace peer review. This was then replaced with proposals that citations be used to "inform" expert review.

But the latest Hefce document says: "We will reconsider whether the benefits of incorporating citation information into the REF outweigh the costs if only a small minority of panels request (it), if the costs are high, or if the equalities implications cannot be effectively mitigated."

Mr Sweeney told Times Higher Education that the funding council was "edging out" of plans to use the data. He said that the costs to institutions of including such a measure could be high. He added that the council had also received an analysis suggesting that their use could disadvantage female staff.

"Equal opportunities is a problem that concerns us, so we are very nervous about the use of citations," he explained. This did not mean that citation data were being ruled out once and for all, he said, but "we will be discussing it closely with the panels before we give it any significant role".

Equally significant for institutions' preparations was the news that there was likely to be further slippage in the timetable for the framework's implementation.

Mr Sweeney said universities could expect a minimum of another year's delay to the current timetable. This would push the REF assessment to 2014 and REF-based funding to 2015.

"It has got to be delayed by a year because we are having a more detailed discussion about impact than we expected," Mr Sweeney said.

The Initial Decisions document contains no significant fresh details on how impact will be assessed: a pilot exercise on the matter is due to conclude in the autumn.

However, it does say that the number of REF panels will now be "30 to 40", instead of the 30 suggested in the initial consultation.

Academics will also have to be contracted to do research by their institutions for their work to be eligible for submission.

Mr Sweeney stressed that Hefce would press ahead with plans to assess impact, despite the warning from the House of Commons Science and Technology Committee last month that finding a fair and effective way of doing so retrospectively was an "insurmountable" problem.

"We will test the methodology and if it doesn't prove adequate then we will reconsider, but we remain confident that the issues can be surmounted," Mr Sweeney said.

Jonathan Adams, director of research evaluation at data-analysis firm Thomson Reuters, said citation analysis should not be written off entirely as a check on expert review. He warned that if it were, the Treasury may demand that "quantitative rigour" be forced back into the process.

"The hare is running and if it (is not caught), a new government may chase it in all sorts of unhelpful directions," he said.

Linda Butler, head of the research evaluation and policy project at the Australian National University and a consultant on various aspects of the REF, agreed that citation data should play a part. But she added that the plan to use citations for individual papers rather than at the subject level was "not robust" and could encourage "game-playing".

zoe.corbyn@tsleducation.com.

You've reached your article limit.

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments