Submissions focus on the weighting of impact in REF

Two key university mission groups whose opinions commonly divide the sector have converged to recommend a 15 per cent weighting for impact in the forthcoming research excellence framework.

The Russell Group of large research-intensive universities and Million+, which represents new universities, have been unveiled as unlikely bedfellows after setting out their positions in the Higher Education Funding Council for England's consultation on the REF.

Hefce has previously suggested that impact should account for 25 per cent of the total score.

The funding council has received 300 or so responses to its consultation. Universities UK advocates a 20 per cent weighting for impact, while the learned societies are deeply divided, and yet more scholars have joined the grass-roots protest calling for impact to be dropped altogether.

The consultation on the REF, which will replace the research assessment exercise as the means of allocating nearly £1.5 billion in annual quality-related research funding in England, closed on 16 December.

Responses show broad support for a system that shifts the assessment's emphasis back to peer review, which is familiar from the RAE. There is also widespread concern that a proposed 25 per cent impact weighting is too high.

This week, senior law scholars and early-career researchers joined those demanding that impact be dropped (see box, page 7).

David Sweeney, director of research at Hefce, said he understood why a lower weighting was being suggested, given the untested nature of measuring impact. "The pilot exercise will test the robustness (of the measure) ... It would be premature to settle on a figure until the outcomes of the pilot are known."

Under the Hefce proposals, 60 per cent of a department's REF score would be based on outputs, 25 per cent on impact and 15 per cent on the research environment.

Proportional responses

But in draft submissions provided to Times Higher Education, many organisations call for a different ratio.

The Russell Group and Million+ both suggest that the weighting for output should be boosted to 70 per cent, with impact cut to 15 per cent - the same as for environment.

However, they have very different reasons for their positions.

The Russell Group argues that Hefce's proposed impact score is too high and will have "potentially destabilising" consequences, adding that 15 per cent should be the "maximum" considered. Million+ worries that pressure to reduce impact would result in a bigger environment component and a "perverse" focus on critical mass in research.

It says that "many of the concerns (about impact) are largely exaggerated" and adds that the "relevance of 'environment' to an outcomes-oriented REF" is the most questionable factor in the proposals.

"Hefce's proposed weighting of 15 per cent (for environment) is the maximum that should be allowed," it says. "A more defensible distribution of weightings" is 70:15:15 for outputs, impact and environment, it argues. The 15 per cent it suggests for impact would be a "minimum", rather than the maximum it represents for the Russell Group.

UUK asks Hefce to give "serious consideration" to the 70:15:15 ratio, but proposes 60:20:20 as its preferred alternative. "Outputs should be the main element of the assessment. We also believe that impact and environment should have the same weighting," it says.

The 70:15:15 ratio is also advocated by the University Alliance, speaking for "research-engaged" institutions.

The 1994 Group, which represents smaller research-intensive universities, says it will reserve its decision on weighting until the results of a Hefce pilot study are available. It does, however, note that "the research environment should be given more weight than the consultation proposes" and it offers a 65:15:20 ratio as a good starting point.

Meanwhile, among learned societies, there are marked differences of opinion on the weight of impact. The British Academy and the Academy of Social Sciences are among those that suggest 15 per cent, but others oppose setting a single figure.

The Royal Society argues that to ensure that no discipline is disadvantaged, individual panels should be left to set their own weightings.

John Pethica, vice-president of the Royal Society, said: "The weightings that are given to different criteria will need to suit individual subject areas. Hefce needs to learn from the pilot studies and allow panels flexibility."

The Royal Astronomical Society joins the University and College Union in arguing that impact should not feature in the REF at all, although it adds that if it is to be used, it should account for no more than 5 per cent of the total score.

The Society of Biology argues that the impact weighting should be 10-15 per cent "at most".

Others, such as the Research and Teaching Group, welcome the inclusion of impact but believe that it should encompass the effect of research on student learning.

As an indication of the perspective of business, the Confederation of British Industry proposes that impact be raised to 30 per cent, while the Association for University Research and Industry Links supports Hefce's 25 per cent figure.

Too much, too soon?

Another theme to emerge from the draft submissions is whether the timetable for implementing the REF should be extended.

The University Alliance argues that "there is a strong case for pushing back the REF by a year". Million+ says such a delay would be "sensible", and the Academy of Social Sciences concurs.

UUK notes that "the timetable set out by Hefce is extremely tight", adding that the timing should be reviewed if key targets for publishing guidance and criteria are not met.

Respondents also express concerns about Hefce's plan to cut the number of subject panels from 67 to 30 - particularly in relation to the six engineering sub-panels, which will create the largest unit of assessment in the REF.

The Russell Group is particularly troubled by Hefce's 10- to 15-year time frame for "high-quality" research impact, which it says is too short. Although it would prefer no time limit, if one is to be imposed, it should be 20 years from when research is undertaken "as an absolute minimum", the group says.

In supporting individual panels determining the appropriate time frame for impact, UUK also highlights doubts about how the "high-quality" nature of research is to be identified.

Also evident in submissions are differences of opinion on plans to cut the number of outputs submitted for assessment from four to three.

Although the proposal is backed by Million+, the Russell Group and the 1994 Group claim that it would reduce the system's "robustness" and "discriminatory power". UUK says that while it may ease the burden on panels, it is unlikely to shrink institutions' workloads.

Another point raised in a number of submissions is precisely how the inclusion of impact in the REF may change researchers' behaviour.

"It will be important to explore the behavioural implications in the pilot," UUK says.

zoe.corbyn@tsleducation.com

THE CAMBRIDGE VIEW: A ROUTE TO 'SECOND-RATE' STATUS

Giving impact a "high weighting" in the research excellence framework would be an "irresponsible" move that would turn "first-rate universities into second-rate companies", the pro vice-chancellor for research at the University of Cambridge has said.

In a synopsis of the university's submission to the REF consultation, which proposes that impact should be valued at 10 per cent, Ian Leslie says that impact with a "low weighting" would encourage institutions to communicate their research, while a high one risked undermining "the most efficient research environment in the world".

"The proposal to measure impact and distribute 25 per cent of quality-related research funding on this basis is neither credible nor responsible," he says, stating that impact is "extremely difficult" to quantify and that such a measure would create undesirable incentives.

He adds that there are some who support impact's inclusion in the REF "because they want to see behavioural change" in universities.

"We support the incentives for institutions to communicate the impact of their research, but we see no case for changing the research that they do ... A low weighting will incentivise this communication, a high weighting will incentivise a shift to research with short-term - and therefore more easily traceable - impact, turning first-rate universities into second-rate companies."

LEARNED FRIENDS AND OTHERS OBJECT

Senior legal academics and early-career scholars have added their voices to the growing calls for the Higher Education Funding Council for England to dump plans to include impact in the research excellence framework.

A total of 43 senior scholars in the field of law and legal theory have written to Times Higher Education to express their concern about the plans.

The letter, they say, is a follow-up to one submitted last month by almost the entire philosophy sub-panel from the 2008 research assessment exercise.

"We have seen the letter written by members of the RAE 2008 philosophy sub-panel on this matter, and we are in agreement with their broad concerns," they write.

They add that in their subject, which might be considered more "applied" than philosophy, it could be just as problematic to measure impact.

In the letter, they enumerate adverse behavioural changes that they fear will result. They also make clear that - like philosophers - legal academics should pursue research for its intrinsic worth and interest.

"Seeking to assess impact is irrational and will do long-term damage to the balance of intellectual life in law schools," they conclude.

The letter comes as a petition launched by early-career researchers reached 500 signatures. The document - which calls for impact to be left out of the REF criteria - was drafted by early-career academics and postgraduate researchers in the humanities and social sciences.

Tara McCormack, lecturer in international politics at the University of Leicester, said the petition should dispel any idea that objections to the REF were from "a few grumpy old male academics keen to defend their research from hoi polloi".