RAE rankings comes under fire

The lack of assessment criteria in the research assessment exercise resulted in "propagandist promotional text" and "opinions stated as fact", according to a leading US academic.

Eileen Gambrill, Hutto Patterson professor of social welfare at the University of California at Berkeley, was a non-UK adviser to the 2001 RAE panel for social policy and administration.

In the article "I am not a rubber stamper" in this month's Journal of Social Work , she says: "Rankings by RAE panel members seem to be made based on vague criteria and surrogates such as 'esteem indicators' (for example papers presented at conference) that may not reflect the quality of research.

"Government bodies responsible for distributing research funds should give careful consideration to how to fairly and accurately evaluate research quality. At least in this area, it seems that such consideration has not occurred."

Professor Gambrill reproduces lengthy correspondence with the Higher Education Funding Council for England, responsible for the RAE, in which she repeatedly asks for detailed criteria. "I reviewed all material provided, as well as on the RAE website, but did not find specific criteria used to review conceptual papers (for example up-to-date, accurate description of alternative views and related research findings) or different kinds of empirical studies."

She gives examples of "fine-sounding accolades" used to describe research, such as: "Theorising inequalities in healthI permeates all the output of this group"; "It combines theoretical insights with sophisticated methodologies"; and "This research has contributed significantly to the evidence base for subsequent regional plans for services to these client groups".

She concludes: "Without a concrete clear description of the evidentiary basis of such claims, such statements serve more as advertisements than as clear descriptions of what has been done to what effect."

She also asks: "To what extent have the publications described in these reports had any impact on enhancing the quality of services provided to clients and attaining hoped-for outcomes? Isn't this a critical question in applied fields such as social policy and social work?"

A Hefce spokesman said: "We have recently announced a major review of the RAE in which a wide range of issues will be considered, including criteria for assessment."

In July, Janet Lewis, director of research at the Joseph Rowntree Foundation and a member of the social policy and social work RAE panel, also complained about how social work was assessed in the RAE. She said in The THES : "Universities and the funding council have not yet found a way of valuing the useful social knowledge that exists in social science departments/research units doing applied work."