High REF scores linked to strong journal impact factors – study

World-leading research identified by UK’s Research Excellence Framework was also found in less well cited journals

May 31, 2023
Judges inspect an arrangement of colourful flowers to illustrate High REF scores linked to strong journal impact factors – study
Source: Getty

Scholarly papers published in journals with high citation rates did better in the UK’s Research Excellence Framework (REF), although 4*-rated research could also be found in less prestigious publications, new research suggests.

Analysing the quality scores of more than 96,000 research papers submitted to the 2021 REF exercise, researchers from the University of Wolverhampton identified a “positive correlation between expert judgements of article quality and average journal citation impact in all fields of science”, according to a new paper in the Springer journal Scientometrics.

That correlation meant that “in all fields, an article in a substantially above average citation impact journal has a reasonable chance of scoring 3* instead of 4*”, explains the study, whose results are likely to revive debate about the usefulness of journal impact factors (JIFs) in assessing research quality.

Critics of the controversial metric, which was originally invented to identify influential publications in a discipline, have long argued that JIFs are an unreliable way to assess research quality. According to the San Francisco Declaration on Research Assessment (Dora) signed by thousands of scientists since 2012, impact factors should not be relied upon in hiring, promotion or funding decisions.

However, the fact that independent REF review panels who are urged not to be influenced by journal reputations often rated highly papers that came from highly-rated journals may strengthen calls to use metrics more extensively in future quality assessments.

However, the study’s lead author, Mike Thelwall, professor of data science at Wolverhampton, said it was important to note the “weak” or “moderate” correlation found in many subjects.

“When it comes to 4* papers, there was a real mix in the journals where it appeared,” Professor Thelwall told Times Higher Education. “You can find 4* research in any journal, even those with very low citation rates,” he added.

Professor Thelwall’s paper, titled “In which fields do higher impact journals publish higher quality articles?”, found the strongest correlations between REF-judged quality and citation-linked excellence in health and life science subjects, and the weakest link in arts and humanities subjects, though there was wide variation within subject areas.

That “lack of a strong correlation between article quality and average journal impact within any fields” which was “never above 0.5 for any unit of assessment, never above 0.42 for any broad field, never above 0.54 for any large narrow field, shows that journal impact is never an accurate indicator of the quality of individual articles”, the paper concludes.

“This result confirms that DORA’s advice ‘Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions’ (DORA, 2020) is empirically valid for all academic fields,” it adds.

While he opposed the use of JIFs to judge the quality of individual papers, this metric could nonetheless prove useful in some contexts, said Professor Thelwall.

“If you’re evaluating large numbers of papers in an academic area, it can be useful to assess quality. We used it in some of our AI research related to the REF which sought to predict the quality of papers – it helped but it wasn’t a strong predictor of quality,” he said.


Register to continue

Why register?

  • Registration is free and only takes a moment
  • Once registered, you can read 3 articles a month
  • Sign up for our newsletter
Please Login or Register to read this article.

Related articles

A push to end the habit of assessing researchers by their publication metrics is gaining momentum. But are journal impact factors really as meaningless as is claimed? And will requiring scientists to describe their various contributions really improve fairness and rigour – or just bureaucracy? Jack Grove reports

9 December

Reader's comments (4)

When I served as a research dean we didn't bother doing any reading of articles. We weighted each article by the 5-yr JIF to come up with a collective score that we then ran simulations around to generate the best mix of articles to submit (based on not just the mean score but the variance of outcomes that could arise). Doing that predicted the REF outcomes with more than 85% accuracy. When I was involved in the Australian equivalent, our panel argued for doing away with individual reading of articles (saving hundreds of thousands of hours of people's time) because the JIF weighting model was very accurate in predicting collective scores (it is less accurate at the individual level but the who exercise is meant to be collective). Needless to say, this was not taken up and millions of dollars of productive time has been wasted since reading articles.
Most of these papers are written by AIs anyway, but don't worry, we have AIs to detect the papers written by AIs. The problem comes when the AIs start plagiarising the AI detection AIs. What we really need is an AI to detect AI plagiarism of AI. That'll sort it.
“in all fields, an article in a substantially above average citation impact journal has a reasonable chance of scoring 3* instead of 4*” - Right. How is this a good thing? I could understand if it said for example that it has a reasonable chance of scoring 3* instead of 2*.
In my discipline (history) I think part of the difference is that the high impact journals are often generalist, with a broader readership across the discipline (and beyond) rather than with a more limited chronological, thematic, or geographical focus of other journals. This is not to say that all articles in generalist journals are good or that good and excellent work isn't published in more specific journals. Rather I think often the difference in scoring and perception of significance lies in the 'framing' of the article in the introduction or opening and the claims it makes to address big/ important debates in one or more areas.


Featured jobs