Get off your high horse

Academics should back a case-study approach to impact in the REF or risk getting far worse, says Claire Donovan

August 25, 2011

For some, the old maxim that "a camel is a horse designed by committee" neatly fits the proposed blueprint for assessing research impact in the research excellence framework. However, there are compelling reasons why it must take this peculiar shape.

Five years ago, I chaired an Australian government committee seeking the optimal method to assess the broad social benefits of academic research. This informed the development of what was an equivalent of the UK's REF.

We found that standard quantitative impact measures - number of patents, spin-off companies, commercialisation income and so on - lacked robustness. They said little about the benefits of work, privileged economic private value over wider public value and had little relevance for basic research, especially in the humanities, arts and social sciences.

One day, my committee secretary phoned me with an idea for a novel metric to capture impact on policy: citations in Hansard. What about counting how often research was mentioned in parliamentary debate? Even better, he went on, whether the research was discussed favourably or not would be a measure of positive or negative policy impact. After a pause, I asked what it would mean if a positive or negative citation was made by government or opposition. The line went quiet. Thankfully, this idea was not mentioned again.

The committee concluded that a lack of robust impact measures made a metrics-only exercise untenable. Australia's previously metrics-bent chief scientist readily accepted a case-study approach.

Since the early 1990s, the research evaluation community has developed a variety of ways to gauge the broad social and economic benefits of research. State-of-the-art methods fuse case studies with robust supporting quantitative and qualitative data.

A similar case-study approach has been proposed in the consultation document on draft panel criteria and working methods for the REF. There are several reasons why this deserves support.

First, scholars in the humanities, arts and social sciences have often been the harshest critics of impact. This is understandable - impact has often been presented as an instrumental economic rationalisation of the value of research. Yet this applies largely to a metrics-only approach; case studies reveal the wider benefits of research. If we disengage with the impact agenda entirely, or reject the principle of impact case studies, a likely alternative will be "one-size-fits-all metrics" that conceal socially and culturally valuable research outcomes.

Second, some natural and social scientists advocate metrics-only impact assessment on the grounds that narratives are "fairy tales" and peer review is "subjective". But this is "impact-lite". Removing the power of narrative explanation and expert judgement will render the evaluation process superficial. "Objective" metrics often gloss over much more than they reveal.

Third, a great deal of research produces social, cultural, environmental and economic benefits that go unrecognised. Impact assessment in the form proposed illuminates this public value. It also strengthens the case for government support for the humanities, arts and social sciences on their own terms.

This is also a cautionary tale, for the Australian committee's recommendations did not come to fruition. With a change of government, a new minister looked at the case-study approach to assessing impact and saw a camel. It was replaced by four streamlined, simple metrics: plant breeders' rights, patents, registered designs and commercialisation income.

The lesson here is that the worst possible response to the consultation is to reject the case-study approach. Of course, there are caveats. The REF methodology is correct in principle but needs fine-tuning. Basic research must remain valued: one solution is for impact to be optional but eligible for rich rewards. And assessments must not stifle impact "in the making" or the efforts of younger research groups.

It is in all our best interests to respond to the REF consultation with constructive suggestions. Otherwise, get the hump and get simple metrics.

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Featured Jobs

Assistant Professorship in Behavioural Science LONDON SCHOOL OF ECONOMICS & POLITICAL SCIENCE LSE
Foundation Partnerships Officer LONDON SCHOOL OF ECONOMICS & POLITICAL SCIENCE LSE

Most Commented

Social media icons

Gabriel Egan laments the narcissistic craving for others’ approval brought on, he says, by the use of social networking websites

James Fryer illustration (8 September 2016)

Some lecturers will rightly encourage forms of student interaction that are impossible for those covering their faces, Eric Heinze argues

University of Oxford students walking on campus

University of Oxford snatches top spot from Caltech in this year’s World University Rankings as Asia’s rise continues

Handwritten essay on table

Universities must pay more attention to the difficulties faced by students, says Daniel Dennehy

Theresa May entering 10 Downing Street, London

The prospect of new grammar schools on the horizon raises big questions for HE, writes Nick Hillman