RAE reviewers accused of going over the top on scores

Concerns have been raised that reviewers for the research assessment exercise have been "too generous" in awarding the top grade.

The results from the 2008 RAE, published on 18 December, said that 17 per cent of all research submitted by more than 50,000 researchers was "world leading" (4*). Some 150 of the 159 institutions included in the exercise were shown to have at least 5 per cent of world-class activity in at least one department.

Initial celebration of the results as a clear sign of the UK's international research excellence has been replaced by complaints about "grade inflation" and warnings that this could affect the distribution of funding.

Observers have expressed concerns about what they say are the "very high" percentages of 4* work in certain subject areas. Some peer-review panels in arts-based subjects, as well as in business and economics, seem to have been more generous in awarding the top grade than those marking sciences or languages.

In communication, culture and media studies, for example, the highest-scoring department saw 65 per cent of its research activity graded 4*. Dance and music have similarly high figures.

However, the top-scoring physics department saw just 25 per cent of its research graded 4*.

Top departments in bioscience-related subjects, where the UK is considered to excel, hover between 30 and 40 per cent for 4* research activity. In languages, the departments with the highest percentages of 4* research activity are grouped in a similar bracket.

"It would have been better if all the subjects had used the top category (4*) less," said Andrew Oswald, professor of economics at the University of Warwick. "The whole point of the categorisation from 1* to 4* was to try to separate out work that was on the edge of winning Nobel prizes, and to get people to understand the difference between good work and absolutely breathtaking, iconoclastic work. I think that a general problem with the RAE is that all panels have put too many people into that 4* box."

Eric Thomas, vice-chancellor of the University of Bristol, said that although the 17 per cent figure for 4* overall was "probably" about right, he was concerned that several panels had been too generous in their interpretation of 4*.

"Some results show more than 50 per cent 4* in some institutions in particular units of assessment," he said. "The grade 4* was deliberately described as world-leading research which I would have thought of as a minority, not a majority," he said.

Departments were graded by 67 subject subpanels moderated by 15 main panels, which also met to ensure consistency in how the grades (of 4* for world-leading research through to 1* for nationally recognised research) were interpreted.

The Higher Education Funding Council for England, which ran the RAE, has said that the results across panels are supposed to be "broadly comparable".

But one panel member, who asked not to be named, agreed that it had not been "too hard" to get a 4*.

"Is it the Nobel prizewinning paper or something that is much more common but nevertheless drives the field forward? ... In the end we decided that it was something that happened to good people every year," he said.

• The RAE's exposure of pockets of research excellence across a range of teaching-intensive institutions could lead to extensive poaching of staff by the more powerful research institutions, the sector was warned this week.

This RAE provided, for the first time, a "research profile" for each department rather than a single summative judgment. The profiles highlighted pockets of excellence in departments that, overall, are not world class.

One senior university executive commented: "It is the other side of the exercise. You identify the very good universities so people want to be in them, but ... any identification of clever people means the market will (naturally) look to them. When we are looking to hire at whatever level, we want someone good so we look around to see where they are."

zoe.corbyn@tsleducation.com.