RAE's non-specialist gambit could have led to blunders, says study

Non-expert judges may not have been up to the task, chess research claims. Zoe Corbyn reports

May 7, 2009

A study into the behaviour of expert chess players has led a team of researchers to make a controversial hypothesis - that those who judged the quality of work submitted to the 2008 research assessment exercise were not qualified for the task.

The academics contacted Times Higher Education last week to report the results of their study into skill levels and behaviour among chess experts.

Their study, published in the journal Cognitive Science, looked at how 24 top chess players performed when they were forced to begin games with unfamiliar moves, rather than the openings they specialise in.

After starting in this way, the performance of the chess players was markedly reduced, as measured by the Elo rating scale used as the international yardstick for chess performance. Players normally rated as grandmasters dropped more than 2,600 places in the rankings, exhibiting performance closer to that of international masters.

But the cognitive psychologists said that the findings from the paper, "Specialisation effect and its influence on memory and problem-solving in expert chess players", do not simply help to explain the mechanisms of learning and expertise. They also have ramifications for the RAE and its replacement, the research excellence framework.

The RAE saw academic peer reviewers judge the quality of research papers outside their specialist areas. Just like the chess players, who floundered outside their comfort zones, the RAE reviewers are also likely to have struggled, the researchers claim.

"(Our research) sheds some worrying light on (the) assumption ... that experts in the panel of each discipline were able to judge the quality of papers when these were outside their domain of specialisation," the researchers said.

"If the results of our experiments generalise to the type of evaluation done in the RAE, then there is room for doubt that the assessors were properly qualified for their task."

The team comprises: Fernand Gobet, professor of cognitive psychology at Brunel University; Peter McLeod of the department of experimental psychology at the University of Oxford; and Merim Bilali? of the department of neuroradiology at Tubingen University in Germany.

The researchers acknowledged that there are obvious differences between evaluating the quality of research papers and finding the best chess move, but added that it is "doubtful" whether the effect of non-specialisation would disappear.

In fact, it could be more pronounced because there are greater differences in specialisation in the academy than among chess players, they said.

Having research papers evaluated by two experts, as was done in the RAE, might "reduce" the effect, but there would be no guarantees if both academics came from outside the domain under scrutiny, they added.

Professor Gobet emphasised that the extrapolation of the research findings to the RAE was not meant to be a joke.

"The implications are very serious for the entire approach," he said.

• The Higher Education Funding Council for England has published the content of more than 2,300 submissions to the 2008 RAE.

Anyone can find out exactly which research papers by which academics in every department were submitted for assessment.

Also available is information about each department's research income, and how they described their environment and "esteem" to RAE reviewers.



Already registered?

Sign in now if you are already registered or a current subscriber. Or subscribe for unrestricted access to our digital editions and iPad and iPhone app.

Have your say

Log in or register to post comments

Featured Jobs


Most Commented

Elderly woman looking up at sky

A recent paper claims that the quality of researchers declines with age. Five senior scientists consider the data and how they’ve contributed through the years

Otto illustration (5 May 2016)

Craig Brandist on the proletarianisation of a profession and how it leads to behaviours that could hobble higher education

Woman tearing up I can't sign

Schools and universities are increasingly looking at how improving personalities can boost social mobility. But in doing so, they may be forced to choose between teaching what is helpful, and what is true, says David Matthews

Eleanor Shakespeare illustration 19 May 2016

Tim Blackman’s vision of higher education for the 21st century is one in which students of varying abilities learn successfully together

Door peephole painted as bomb ready to explode

It’s time to use technology to detect potential threats and worry less about outdated ideas of privacy, says Ron Iphofen