When the results of the research assessment exercise 2008 were announced, concerns were raised about the scores awarded by different sub-panels, with UK institutions appearing to perform more strongly in media studies, for example, than in physics.
So should sub-panel scores in the forthcoming research excellence framework be "normalised" to prevent a repeat of such comparisons?
Dame Julia Higgins, emeritus professor of polymer science at Imperial College London and former head of its faculty of engineering, this week called for some "serious thinking" on the issue from the Higher Education Funding Council for England.
The REF will replace the RAE as the means of allocating nearly £2 billion in research funding annually from 2014, and is currently the subject of a sector-wide consultation.
Dame Julia is well qualified to comment on the process, having been involved in almost every RAE since the assessments began.
She chaired the main panel covering chemistry, physics and earth sciences in RAE 2008, and is a stalwart of the academy, having held such positions as foreign secretary and vice-president of the Royal Society and chair of the Engineering and Physical Sciences Research Council.
Dame Julia said Hefce did not want to face up to not having a basis for comparing subjects, but said it needed to think hard about how comparisons were going to work in the forthcoming REF and engage in some "frank discussion" on the topic.
"We have to have an open discussion about whether we are expecting to compare subjects and on what basis, and, if we are not expecting to, whether it would be much better to use a normal distribution," she told Times Higher Education.
She urged the funding council to iron out the problems now, rather than wait until the REF began, warning against a repeat of RAE 2008 when sub-panels made their assessments and main panels then exerted "pressure" to bring the results into line with each other.
Myth of superiority
For Dame Julia, there is "no intellectual basis" for making such comparisons, and the myth that some subjects are stronger than others should not be perpetuated.
In RAE 2008, a two-tier panel structure was employed, with 67 sub-panels working under the guidance of 15 main panels. She said this had a greater "normalising" effect than systems used in previous exercises, but there were still variances between subjects.
These discrepancies led to accusations that sub-panels in some arts-based subjects had given themselves an easy ride and that science subjects had sold themselves short.
In physics, 25 per cent of the best-rated department's research was judged to be "world leading" (4*), whereas in media studies 65 per cent of the top department's research received the highest marks. On average, 15 per cent of work submitted to the biological sciences panel was judged to be 4*, compared with per cent in economics.
"The real problem is that people will insist on using (the results of the REF) to try to make comparisons between subjects," Dame Julia said.
"They will say media studies is better in this country than physics when everybody knows perfectly well it isn't ... Differences between the averages for the panels do not mean one is worse than the other, it just means people have interpreted the grades slightly differently."
She said the only way to avoid meaningless comparisons in the future was to normalise distribution, making the proportion of work that could be expected to fall into each quality category the same. She said the panel chairs had told Hefce during RAE 2008 that it did not have "any basis" for comparing subjects, "but it refused from the outset to consider a normal distribution".
"What Hefce should be seriously thinking about for the REF is specifying how the distribution will look," she added.
The Hefce consultation, which is being wound up this month, proposes measures to ensure greater consistency across panels, ranging from more central guidance to a reduction in the number of sub-panels to 30, but it does not mention normalisation.
Peter Golding, chair of the media studies panel in RAE 2008, agreed that it was important to ensure that the REF results were not interpreted as indicating that one subject was better than another.
But he said: "Whether normalisation will achieve this depends on how it is done, not least if there are very diverse, composite panels as envisaged in the REF consultation."
Dame Julia also praised the re-introduction of peer review in the REF, and said she had "no particular problem" with the proposed 25 per cent weighting for impact, nor the reduction in the number of sub-panels. She supported the availability of citation data for reviewers, but raised concerns about the implications for interdisciplinary subjects, where studies are published in a diverse range of journals.