REF review ‘will focus on diverse outputs and research culture’

Minister’s attack on academic publication culture suggests a move towards more holistic and team-based assessments of excellence, say experts

October 26, 2020
Sheep are being judged in a show ring
Source: Getty

Plans to reform the UK’s research excellence framework (REF) may lead to a radically different exercise in which research culture is valued as highly as outstanding publications, a policy expert has predicted.

Announcing a review of the REF, which is used to distribute about £2 billion in research funding annually, science minister Amanda Solloway focused on the “pressure to publish in particular venues”, which “wrongly suggests that where you publish something is more important than what you say”. She noted that 97 per cent of outputs in the 2014 REF were “text based” and mainly journal papers.

That criticism suggested that the REF review may seek to broaden the type of outputs submitted by researchers, explained James Wilsdon, Digital Science professor of research policy at the University of Sheffield, who expected to see a push to include research datasets, thinktank policy papers, exhibitions and other diverse outputs in the next audit.

“Despite the current guidance that outputs should draw from a diverse set of venues, there is still definitely a strong weighting towards articles in journals with high impact factors,” said Professor Wilsdon, who added that “publication in a prestigious journal gives some reassurance of quality”. “If you’re entering something into a competitive process, you want to be sure that output is a banker – a dataset on Figshare may be more helpful for science, but it doesn’t give you that external assurance,” he added.

One way to address this issue would be to reduce the importance attached to outputs, which have a 60 per cent weighting when peer assessments are made by REF panels, said Professor Wilsdon. Panels also examine the impact of research, which has a 25 per cent weighting, while the environment in which research was conducted accounts for 15 per cent.

“If you had an even weighting of outputs, impact and environment, the REF would start to look like a very different and interesting exercise,” said Professor Wilsdon.

An increased emphasis on environment would move the exercise towards a broader consideration of research culture, team science and research integrity – issues championed by the new chief executive of UK Research and Innovation, Dame Ottoline Leyser.

Professor Wilsdon said Ms Solloway’s speech indicated a desire to complete the decoupling of outputs from individuals recommended by the 2016 Stern review, which led to the requirement for four outputs for every submitted full-time equivalent researcher in 2014 being reduced to an average of 2.5, and a minimum of one output per researcher, in 2021. “If you slice the knot between people and outputs completely and just look at units, that will encourage a more holistic view of research culture,” explained Professor Wilsdon.

David Sweeney, executive chair of Research England, which will oversee the REF review, said that he was “glad that the sector and its partners will have the opportunity to discuss what they really think is excellent research”.

“We are delighted by the work done by our highly talented academics, but we look forward to capturing the support that is around them which helps them to deliver the outcomes of which the UK is so proud,” he added.

However, Tim Softley, pro vice-chancellor for research and knowledge transfer at the University of Birmingham, said he would be concerned about any plan to reduce the importance attached to research published in highly cited journals.

Speaking at a Higher Education Policy Institute event, Professor Softley said that while research culture was important, “we do seem to be getting into the habit of saying there is something wrong about going after excellence”.

“There is nothing wrong with trying to publish a high-quality publication in a prominent journal where lots of people will read it,” said Professor Softley.

Register to continue

Why register?

  • Registration is free and only takes a moment
  • Once registered, you can read 3 articles a month
  • Sign up for our newsletter
Please Login or Register to read this article.

Related articles

Reader's comments (5)

"However, Tim Softley, pro vice-chancellor for research and knowledge transfer at the University of Birmingham, said he would be concerned about any plan to reduce the importance attached to research published in highly cited journals." Of course he would be. Senior management is afraid of loosing the stick with which they have got used to wack the pesky academics with (i.e. individual performance tragets to manage our "excellence" derived from the REF). “There is nothing wrong with trying to publish a high-quality publication in a prominent journal where lots of people will read it,” said Professor Softley" Sorry, that may be true in science, but in my field rarely anyone reads the "top journals" (read US journals) as they have become so technical and obscure that these highly "excellent" articles increasingly have lost their scholarly relevance. In my field, the really exciting and novel stuff is published in niche and specialist journals not the mainstream ones. But hey, let us continue with this one-size-fits-all REF rubbish (e.g. four publications are nothing to talk about in some fields but a great achievement in others).
The problem we need to fix first and foremost is how we define excellence. The peer review model is, as many have previously pointed out, flawed and broken. The model of academic publishing in specialised journals (whether high impact or not), among many other deficiencies pushes knowledge into siloes, resulting in duplication and mutually unintelligible language that hampers interdisciplinarity. Starting from a definition of excellence that better measures social value would lead to a very different kind of assessment and, I would argue, the natural downgrading of the journal article as the prime referent of research excellence.
Having a positive research culture and good publication record are not mutually exclusive. I don't believe my colleagues and I are suddenly going to stop bothering about doing good science because the focus of REF has shifted. A positive research culture could lead to better publications as team science seems to work well for producing high quality, innovative scientific outputs.
As I have said before and will say it again - it doesn't matter what policies REF makes because mock REF conducted privately by all universities will be an opportunity for REF reviewers to subvert these policies with impunity.
" mock REF conducted privately by all universities" - this is wastage of tax payers money of the highest order. Can people not see it??!! What purpose does it serve? Trying to second guess that the panel might do..utter nonsense. The only purpose is either to put more pressure on staff and/or to decide what and how much to buy in to submit to the REF. Mock REF reviews are now also used internally by universities for progression and promotion decisions too and god knows with what levels of transparency and oversight. All of this game playing must end. It only encourages short-termism which has made academia sadly what it is today. Academics have been teaching and researching for centuries before the alphabet soup was invented and as I have said many times, the best universities are inspite of any REF not because of it.