Senior university officials have called for clearer guidance on what ratings in the 2008 research assessment exercise will mean for academics as they prepare for the final criteria to be unveiled next month.
They have also raised concerns that RAE panels for different subject areas will adopt inconsistent approaches to key issues — such as applied research and young researchers — in the exercise.
The higher education funding councils will publish the final criteria in the first week of February, along with an RAE "handbook" offering advice on how the assessment will be undertaken. University departments submitted to the exercise will be classified according to four research ratings, from 1* to 4*, with 4* work defined as world-leading research.
But Mike Cruise, pro vice-chancellor for research at Birmingham University, said: "We would like to see more clarity about what the 4* grade really means.
"World-leading can mean one person in the world or something different and we do need more guidance. It would be very unfortunate if a particular academic area were deemed to be failing in some way because only a small percentage are grade 4*, when that was the result of an arcane interpretation of what the funding
council must have meant when they wrote ‘world-leading’."
Ian Leslie, pro vice-chancellor for research at Cambridge University, said: "We are still a little concerned about the inconsistencies. It looks like there are going to be 60 different RAEs. We won’t know what the funding implications are until we get the grant letter in March 2008."
Universities felt they were "operating in a fog", he said, claiming: "Each panel’s definitions are a little bit different."
Graham Dockray, pro vicechancellor at Liverpool University, warned against panels trying to be too consistent: "[Main] panels are ultimately responsible for the gradings produced, but in seeking consistency, they might have disregarded the better judgments of some sub-panels where they are seeking to address subject specifics."
Meanwhile, some academics judging research are considering carrying out "dry runs" to test criteria ahead of 2008.
Sir Leszek Borysiewicz, chair of main panel A (clinical studies), said: "We have been talking about what format a dry run will take but not exactly when. If you do it too early, some institutions will be at an advantage. Would you make it all public afterwards or not use data from this exercise? We will be looking at the feasibility of this."
Leader, page 14