Are research output measures more worthy than critical review?

Citation in high-impact journals valued more than scholarly assessment

June 12, 2014

Source: Alamy

Paper trail: journal impact factors are a straightforward way to signal quality

Academics’ desire to be judged on the basis of their publication in high-impact journals indicates their lack of faith in peer review panels’ ability to distinguish genuine scientific excellence, a report suggests.

The report, Evidence for Excellence: Has the Signal Overtaken the Substance?, written by Jonathan Adams, chief scientist at Digital Science, and Karen Gurney, a consultant to the firm, analyses trends in the outputs submitted to the past three research assessment exercises.

It finds that the proportion of submissions made up by journal articles has increased significantly, from 62 per cent in 1996 to 75 per cent in 2008. This reflects significant declines in the number of monographs submitted by social scientists and in the number of conference papers submitted by engineers – although books and chapters remain popular in the arts and humanities. Rather than a “massive cultural shift”, the report says this “looks much more like a change in behaviour, not in what was being written but in what was being offered for assessment”.

The report suggests the reason is that journal impact factors provide academics with a simple and widely used way to signal the quality of papers of theirs that have appeared in those journals, whereas “similar databases are still only superficial for conference proceedings and books”.

“Perhaps the change in behaviour…is evidence that numbers inexorably overcome real cultural preferences,” the report, which was published on 9 June, says.

An analysis of the 2008 RAE reveals that 14 high-impact journals each accounted for more than 500 of the nearly 81,000 articles submitted. Three journals with particularly high impact factors – Nature, The Lancet and Science – accounted for many more RAE submissions than the number of eligible outputs they contained, indicating that many co-authored papers had been submitted by more than one institution. One Nature paper had been submitted by all 12 UK institutions that had authors who had contributed to it.

Although 500 eligible UK Nature papers were not submitted to the RAE, 418 of the total 1,510 submissions of outputs from the journal were not UK-authored papers. These were either papers by academics recruited from abroad or “ephemera”, such as letters or editorials, which were often not cited at all.

“It might seem to make obvious sense for researchers to choose to submit papers from journals that had particularly high impact,” the report says. But it notes that the review panels – including those for the 2014 research excellence framework, whose results will be known in December – are barred from considering journal impact factors and are supposed to base their assessments of outputs on reading them.

Impact factors are an average of the number of citations that papers in the journal receive over a certain period. But the report points out that the figure is skewed by small numbers of highly cited papers, meaning that most papers garner fewer citations. Yet some academics submitted low-cited papers in high-impact journals in preference to more highly cited papers published in lower ranking journals.

“The real substance of what academics thought was the best marker of research excellence was displaced for review purposes by outputs that gave the simplest signal of achievement,” the report says.

“The kudos of the well-cited journal was a marketing signal outweighing the individual item.”

Times Higher Education free 30-day trial

Please login or register to read this article

Register to continue

Get a month's unlimited access to THE content online. Just register and complete your career summary.

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments

Reader's comments (3)

We should bear in mind that academics are not necessarily going to submit what they think is their best work and perhaps not even what they think the REF panel will think is best. They mayl submit what will be best judged by the internal panels that screen the work and assign a star ranking to it for internal management purposes. That will depend then on what criteria management thinks is best.
Thank you David. I have been smiling again and again over how much of the travesty of the moment is captured in your comment. It does get worse, when the academic in question does not know who the evaluator (called by one good colleague "God") is, or why he gave a 1,2,3 or 4 to any given item submitted. It was apparently a single individual in some cases, judging… well, employability of the staff. Terse. Then, the intermediaries (those handling money both to "God" and to themselves in order to prepare the academic submission, which of course they do not understand) started calling everyone to meetings, asking them to comment how they would improve… their performance!!!! I do not have to tell you what happens when an academic starts questioning what they mean. Monty Python movies are no longer humorous, they have become documentaries of academic life in Britain.
This kind of nonsense really has to stop. In the pre-internet days, it was important to get your paper into Nature because most libraries subscribed to it (so your article would be widely read). But today, all articles are effectively equally discoverable, so there's no longer any good reason to want to get your article in Nature (apart from this silly obsession with impact factors). If it's a good article, it's a good article - no matter where it's published. If it's a poor article, it doesn't get any better simply because it's been published in Nature!

Have your say

Log in or register to post comments