REF 2021: Leaders say increased impact weighting ‘justified’

REF supremo David Sweeney and Manchester research lead Colette Fagan hit back at criticisms of measuring societal and economic contributions

May 12, 2022
Gold boxing gloves
Source: iStock

UK research leaders have defended the increased weighting of impact to decide scores in the Research Excellence Framework amid concerns that it unfairly favours scientific subjects and research-intensive universities.

In a webinar organised by Times Higher Education to analyse the results of the 2021 REF, David Sweeney, executive chair of Research England, which runs the UK-wide audit of research, said he was pleased that researchers from all disciplines were focused on demonstrating the social and economic contributions of their work.

“I want the UK to say that every discipline is focused, to a considerable part, on research activity and sharing the fruits of it with partners [in order to] help every citizen of our country,” said Mr Sweeney. “I think we can do that.”

Mr Sweeney criticised the “crude metrics” used by other countries to assess impact – an approach that worked for only a “few disciplines”, he said. The UK’s narrative approach to impact, via case studies, delivered a “richness” to assessment not found elsewhere, he continued.

His comments come after the results of the latest REF show that medical institutions excelled on impact, which accounted for 25 per cent of scores, up from 20 per cent in 2014. Arts, humanities and social science disciplines scored less highly on research impact, while impact scores tracked closely to the ratings of outputs – outcomes that helped scientific subjects, as well as older, more science-intensive universities, to score more highly overall.

Colette Fagan, vice-president for research at the University of Manchester, told the event that she believed there was a “bit of room for moderation” to ensure that the impact of research in the arts, humanities and social sciences was rated more in line with science, technology, engineering and mathematics (STEM) research.

However, Professor Fagan said, it was a myth that it was impossible to demonstrate the impact of non-STEM research, even if it was “less linear” than the typical “invention to market” impact often found for life science research.

“When I started at Manchester in 2014…I thought it would easier to evidence the impact in panel A [medicine, health and life sciences] and B [physical sciences, engineering and mathematics] subjects, but it was a different kind of impact,” she said.

“I am less concerned about panel A is higher than panel B for impact – I’m more interested in looking at the impact cases and stories,” which, she believed, were the “most exciting” results of the exercise.

Asked whether the impact scores merely repeated the score for outputs – given their close correlation – Professor Fagan said she would “expect good research to produce good impact”, adding that “one would expect some correlation as we are talking about research”.

However, impact had “shifted behaviour” and asked researchers to “think what their research ambition and agenda is, and what is the range of aims that a project will deliver”, she said.

This year’s REF has introduced a number of changes, including a requirement for all research-active staff to enter at least one output into the exercise, whereas the 2014 exercise allowed institutions to select researchers, who were required to submit at least four outputs.

Asked if the 2021 REF had been less onerous than the 2014 exercise, Professor Fagan said “some elements have been easier” but overall, it had not been any less time-consuming. “We have shifted from looking at individuals to selecting based on the portfolio of a unit – it is worth doing, but it has not shifted the burden,” she added.

In future exercises, Research England should consider changing the requirements for the environment aspect of the REF – worth 15 per cent of assessment – in which units of assessment were asked to write narratives about how they support a positive research culture. “The environment statement can be very time-consuming [considering] the proportion of weighting that it gets,” Professor Fagan said.

Register to continue

Why register?

  • Registration is free and only takes a moment
  • Once registered, you can read 3 articles a month
  • Sign up for our newsletter
Please Login or Register to read this article.

Related articles

Reader's comments (1)

How is the impact measure objective and fair if it largely rests on an element of chance in some disciplines and also on the greater voice some institutions have due to their history and brand. For example, 54% of Surrey Law School's research outputs (actual research) were rated 4*, yet Oxford University’s Law Faculty had only 35% of its research outputs rated 4*, yet Surrey gets an overall rank of 43 for law, while Oxford ranked 29 places higher at 14th place. Notwithstanding Surrey Law was ranked 6th on its actual percentage of 4* research produced, its lack of influence and the lack of invitations to give evidence in parliament meant a good portion of its higher quality research was ignored by lawmakers. While Oxford had greater impact with largely 3* and 2* research outputs. Impact should be reduced to 10% for the next REF and environment to reduced to 5%. Big old brands will always outcompete smaller institutions on environment due to their very large PhD programmes. It is best to focus on the “actual” research produced rather than impact especially for subjects such as law where impact is contingent on an issue arising in society and upon the lawmakers deciding it needs reforming and then upon the chance of being the one that gets listened to.


Featured jobs