RAE faces bias accusation

December 3, 1999

The last research assessment exercise may have been biased in favour of departments with members on assessment panels, research suggests.

Chris Roberts, a senior lecturer in medical statistics at the University of Manchester, has examined data from the past two RAEs using statistical modelling. He has found evidence showing that departments benefited from having a member on a panel.

He argues that bias could occur even though panel members were expected to withdraw during discussion of their own institution.

"A panel member coming from a particular department might be perceived by other panel members as an indication of quality of their department," the report states. "Informal discussion between panel members may have led to greater familiarity with the research activity of each other's departments. Such factors could influence judgement of national or international excellence."

Concerns about bias were raised at the 1996 exercise by heads of environmental science, who wrote to the Higher Education Funding Council for England complaining that their panel was dominated by earth scientists and was not qualified to assess environmental science.

Only 18 per cent of environmental science departments later achieved grades four or five, compared with 52 per cent of earth science departments.

Dr Roberts, who modelled data including outcome rating, panel membership by participating department, research income and numbers of research students or publications submitted, suggests that future RAEs should either select departments randomly for representation on panels or exclude all academics from participating departments.

He concludes: "Given its serious methodological flaws, the 1996 RAE can hardly be described as research of national, never mind international, excellence."

John Rogers, RAE manager for the HEFCE, said: "There is a long way between showing statistical correlation between RAE success and membership of an RAE panel and proving a causal link. I am happy that there is no way in which being a panel member can bias the position of an institution."

He said the HEFCE was strengthening transparency of its processes all the time to ensure people were happy with the way selection of panel members was made.

You've reached your article limit.

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Most Commented

James Fryer illustration (27 July 2017)

It is not Luddism to be cautious about destroying an academic publishing industry that has served us well, says Marilyn Deegan

Jeffrey Beall, associate professor and librarian at the University of Colorado Denver

Creator of controversial predatory journals blacklist says some peers are failing to warn of dangers of disreputable publishers

Hand squeezing stress ball
Working 55 hours per week, the loss of research periods, slashed pensions, increased bureaucracy, tiny budgets and declining standards have finally forced Michael Edwards out
Kayaker and jet skiiers

Nazima Kadir’s social circle reveals a range of alternative careers for would-be scholars, and often with better rewards than academia

hole in ground

‘Drastic action’ required to fix multibillion-pound shortfall in Universities Superannuation Scheme, expert warns