Quarter of citations in top journals ‘wrong or misleading’

Forensic analysis of citations within leading scientific periodicals reveals alarming lack of rigour in academic referencing

October 16, 2020
Small print

One in four references found within leading scientific journals are incorrect or misleading or fail to provide evidence for the claims made by authors, says a new study on academic “quotation errors”.

In a paper published in Proceedings of the Royal Society A, researchers from the US and South Korea analysed some 250 citations taken evenly from five highly prestigious science journals to check whether the references supported the statements made by authors.

While three-quarters of references in the sample taken from Science, Science Advances, Nature, Nature Communications and Proceedings of the National Academy of Sciences were accurate, 25 per cent were deemed not to substantiate the points raised by authors, says the paper, titled “Quotation errors in general science journals”.

About half of faulty citations (12.9 per cent of all citations in the study) were assessed as “impossible to substantiate”, where authors put forward “citations that could not possibly be substantiated by an outside reference”, such as when their own missing methods section was compared to methods sections used by other experiments.

“Sometimes the entire methods section of a paper would be one sentence, explaining how the authors used the same methods used in another paper,” explained Aaron Cumberledge, from the Daegu Gyeongbuk Institute of Science and Technology in South Korea, who co-authored the paper with Neal Smith, from Florida-based AdventHealth University.

“But when you looked closely, it was very difficult to see if they had followed that paper’s methods entirely – there might have been one crucial variant that changed, but it was impossible to tell,” Mr Cumberledge told Times Higher Education.

Some 8.5 per cent of the citations analysed, which were randomly chosen from 500 articles that included a total of 26,344 references, were found to be “unsubstantiated” and 3.6 per cent were judged “partially unsubstantiated” when the error was deemed minor and the general proposition of the quotation was supported by the citation.

In some cases, papers included citations to papers that appeared to contradict their own propositions or failed to provide evidence for them, explained Mr Cumberledge, who teaches science writing at his institution’s College of Transdisciplinary Studies.

Even when the two researchers were able to substantiate citations, the imprecise nature of many references made the checking process hugely time-consuming, particularly as page numbers were generally not included, he added. “It is really difficult to verify things when the only reference is to an old edition of a medical textbook which is 1,000 pages long,” said Mr Cumberledge, who added: “In that case, we managed to substantiate the reference but it was hard work.”

While it was “unfeasible” for editors or reviewers to check all citations for accuracy given the volume of work involved – it took two months for the study’s authors to check 500 citations – Mr Cumberledge called on authors to redouble their efforts to make their citations more precise and verifiable.

By doing so, abusive or unprofessional citation practices, such as spurious citations, in which authors unnecessarily cite the work of colleagues or friends, who, in turn, cite them back, would become much easier to detect, he said.

“I could see some people saying we are being overly fastidious about this, but it would take so little effort to do these citations properly and the science would be so much more solid and evidence-backed.”


Register to continue

Why register?

  • Registration is free and only takes a moment
  • Once registered, you can read 3 articles a month
  • Sign up for our newsletter
Please Login or Register to read this article.

Related articles

Reader's comments (1)

If I had £1 for every time an article that cites me has done so incorrectly or without sufficient care - often entirely misrepresenting me - I wouldn't need to go to work! And in some cases such poor citing appears in theses that I've been asked to examine - and that the author must know I would be reading. It's unfathomable. One approach would be for editors to send copies of a submitted article to as many of the cited authors that are still alive and available, asking them simply to check for accuracy. But in the social sciences - my discipline - where journal editors typically do the job as an aside, with little or no payment, and minimal clerical support, that would be far too time-consuming in practice.