Experts voice concern over REF complexity

Opinions are already divided over the working of the new research assessment system, writes Zoe Corbyn. Debate on the new system to replace the current research assessment exercise began in earnest this week, with experts saying it could become just as complicated as the current system.

Details of the new system - the research excellence framework - to replace the current RAE were released on November 22 by the Higher Education Funding Council for England and are currently open to consultation.

Under the new system, research quality in science subjects will be assessed by drawing up profiles for universities across six large subject areas. These will be based on the number of times academics' papers are cited by others above the average rate in a field. Overseen by peer panels, the data will be combined with metrics on research income and students numbers to determine the allocation of £1.4 billion in research funding starting from 2010.

But while a less onerous and bureaucratic funding assessment is behind the move, Jonathan Adams, director of the data analysis firm Evidence Ltd, warned that the proposed system would require "an awful lot of data processing and checking" both by universities and the funding council.

"People will endeavour to make it less onerous than the RAE, but I think that at the moment they are being very optimistic about what they think is going to be simplified," he told The Times Higher .

He added that the new system could potentially become "very complicated" if the sector started to demand that lots of "correction factors" were introduced to the raw citation data to deal with perceived inequalities, such as applied research being under-cited in comparison to basic research.

"The big risk is that it could become extremely opaque," he said, adding that corrections would be better left to peer panels to judge.

Bahram Bekhradnia, director of the Higher Education Policy Institute, said the new system would have "hugely damaging" effects on behaviour.

He said that as academics and their managers realised that they would be able to do little to increase citation counts, attentions would turn to how to increase research income.

"While attracting more industrial money might be desirable, going for more and more research council grants will not be ... They are time-consuming to apply for and chances of success are low," he told The Times Higher .

Speaking at a conference to discuss the proposals, organised by Universities UK this week, Rama Thirunamachandran, director of research and knowledge transfer at Hefce, said whatever was chosen would drive behaviour.

Also speaking at the UUK conference, Martin Rees, president of the Royal Society, welcomed the new round of consultation and said the society had formed an expert group to come to a view.

Speaking personally, he said that the system must be informed by expert interpretation, "but if one is going to use bibliometrics this is a very good start".

The importance of peer oversight was also stressed by Dr Adams. "Indicators are a very useful aid to research management, but they should never be used in isolation. You need to put that information in front of an informed expert group," he said.

Other key concerns are likely to focus on how to equalise the system for academic career breaks, young researchers and deciding whether the subject groups are too large.

"I think (such large groups) are likely to run the risk of not fully recognising the nuances," said Dr Adams.

Dr Bekhradnia said: "It will be OK for funding purposes, but is far too coarse-grained to provide any useful management information of the sort that the RAE at present provides (strengths and weaknesses of a university or a faculty, for example)."

zoe.corbyn@thes.co.uk

Rae: next steps

Paralleling the start of the debate on the research excellence framework, the final deadline for universities to make submissions to the current research assessment exercise passed this week.

According to Rama Thirunamachandran at the Higher Education Funding Council for England, only about one quarter of universities had dared to press the "send" button on their research assessment submissions by the morning of November .

From January next year, 67 subpanels and 15 main panels will begin a series of meetings to assess submissions and published outputs.

The aim will be to produce a "graded profile" for each university entering a unit of assessment.

This profile will show the percentage of research activity judged to fall into five grades: 4* (top quality), 3*, 2*, 1* and unclassified (below national standard).

The results will be published in December 2008. How it will be translate into research funding for 2009-10 will be fully revealed only in the recurrent grant announcement, which is due in March 2009.