The last research assessment exercise may have been biased in favour of departments with members on assessment panels, research suggests.
Chris Roberts, a senior lecturer in medical statistics at the University of Manchester, has examined data from the past two RAEs using statistical modelling. He has found evidence showing that departments benefited from having a member on a panel.
He argues that bias could occur even though panel members were expected to withdraw during discussion of their own institution.
"A panel member coming from a particular department might be perceived by other panel members as an indication of quality of their department," the report states. "Informal discussion between panel members may have led to greater familiarity with the research activity of each other's departments. Such factors could influence judgement of national or international excellence."
Concerns about bias were raised at the 1996 exercise by heads of environmental science, who wrote to the Higher Education Funding Council for England complaining that their panel was dominated by earth scientists and was not qualified to assess environmental science.
Only 18 per cent of environmental science departments later achieved grades four or five, compared with 52 per cent of earth science departments.
Dr Roberts, who modelled data including outcome rating, panel membership by participating department, research income and numbers of research students or publications submitted, suggests that future RAEs should either select departments randomly for representation on panels or exclude all academics from participating departments.
He concludes: "Given its serious methodological flaws, the 1996 RAE can hardly be described as research of national, never mind international, excellence."
John Rogers, RAE manager for the HEFCE, said: "There is a long way between showing statistical correlation between RAE success and membership of an RAE panel and proving a causal link. I am happy that there is no way in which being a panel member can bias the position of an institution."
He said the HEFCE was strengthening transparency of its processes all the time to ensure people were happy with the way selection of panel members was made.