Do we need harsher sanctions for research misconduct?

Listing a hallucinated artificial intelligence (AI)-generated reference is the “equivalent of doing 3mph above the speed limit. It’s a speeding ticket [but] it shouldn’t mean the death penalty.”

That is the view of Jim Miller, professor of economics at Smith College, a liberal arts institution in Massachusetts. It is not that he is indifferent to academic wrongdoing: he agrees that “people should think less of authors who include AI-generated references”. However, relying on a chatbot to generate references is “not a sign that you’re a failed or deceptive scholar. Mistakes happen,” he told Times Higher Education.

This perspective, however, puts Miller violently at odds with many of his colleagues. Having articulated his view during a recent debate on X about the decision of the physics preprint server arXiv to implement an immediate one-year ban for authors submitting work containing hallucinated refences, Miller was pilloried. Among other things, he was accused of being a “fraud”, “dishonest” and a “pathetic loser” who represented everything bad about his “bullshit discipline”. One critic even suggested Miller’s tenure should be reviewed and pledged to make a formal complaint to his employer.

That unusually fierce war of words perhaps reflects a wider schism in academia about how harshly breaches of ethical norms should be punished. Some believe that permissive standards that allow “mistakes” to be passed over with a correction or, in rare cases, a retraction, have engendered a culture of ever more egregious cheating, fuelled by misaligned incentives in an ever-expanding academia and the rise of paper mills and, now, AI.

“When everyone in the field knew each other and reputations really mattered, relying on retractions [to punish transgressions] made sense,” said Stephen Vainker, a teacher and independent researcher who has called out suspected hallucinated references in several education research journals. “Using AI to write papers or references would have been seen as a stain on your reputation. But I’m not sure many authors really fear getting caught [any more].”

Source: 

Boris Zhitkov/Getty Images (edited)

It is fair to say that arXiv is not the only actor in the scholarly ecosystem leaning towards the view that relying on traditional sanctions is no longer enough to discourage poor practice. Several countries in Asia are set to impose sanctions on those discovered to have violated integrity rules. Vietnam’s Ministry of Science and Technology, for instance, has recently begun requiring organisations to implement rules against research misconduct. Its recommended punishments, according to Retraction Watch, include funding clawbacks and bans from future projects.

India’s leading research agency, the Anusandhan National Research Foundation (ANRF), will soon require academics to declare any retractions of papers when applying for funding, while the country’s domestic ranking of universities, the National Institutional Ranking Framework, will mark down institutions deemed retraction hotspots. In China, several senior scientists have been sacked by their universities after allegations raised by a blogger. And in the US, two attorneys were suspended from practising in a major appeals court for sixth months after they repeatedly filed briefs containing hallucinated references to previous cases.

But is there a risk that sanctions could swing from being too lenient to too draconian? Miller thinks so. “In economics, papers usually have numerous co-authors and will be passed back and forth several times, so errors can creep in,” he reflects. And when it comes to hallucinated references, that peril is especially pronounced given the ubiquity of AI in computer software. “For some…authors, their time is worth thousands of dollars an hour. But we’re expecting every author to check every citation or risk a publishing ban,” said Miller.

That view was endorsed by another economics professor, Eric Rasmusen of Indiana University, on X: “There are a lot of bad scholars who are pedantic. It all depends on the context. If your discovery is correct, a [referencing mistake] is no big deal. In fact, if you’re good, you shouldn’t be wasting your talent on getting the page numbers of your reference right.”

Having recently suffered a stroke, Miller’s own sense of the ease with which genuine mistakes can arise has increased. “Now I’ve lost some functionality in my dominant hand, I can see how you could perhaps think you’ve hit ‘delete’ on a flawed citation, but the keyboard doesn’t register it,” he said.

He also agrees with Rasmusen that hallucinated references figure very low in the hierarchy of citation malpractice. “As a student, I used to edit the Stanford Law Review and would spend days checking citations to ensure they were right. Citations are important to get right, but hallucinated references are so obviously wrong that they don’t damage the literature…If a citation is used to claim one thing but it says a different thing, that is much worse.”

Source: 

Boris Zhitkov/Getty Images (edited)

David Sanders, a biologist at Purdue University in Indiana best known for his research integrity work, takes an altogether more robust view, however. Hallucinated references might represent misconduct in themselves if they are repeated and show “intentionality or recklessness”, he argued. Moreover, they are an “easy screening mechanism for identifying poorly written and poorly edited articles. If the authors can’t be bothered to check their references (which is so easy nowadays), how can we trust them to properly scrutinise their data?” he asked.

For that reason, hallucinations merit not only publishing bans but also retractions, Sanders believes – even if they are unintentional: “Retractions are not only for misconduct. Serious errors that undermine the reliability of the data or their interpretation are sufficient reasons for retraction. The excuse that retraction is unfair because the ‘major conclusions’ are ‘unaffected’ is untenable. The trust that is a condition for relying on the contents of an article has been irreparably undermined.”

Vainker agrees. For him, citing hallucinated works “says you’re just throwing in references without reading the work”. And, for that, a one-year publication ban “should be an absolute minimum. It’s lenient, if anything.”

He also agrees with Sanders that hallucinated references usually indicate far more heinous wrongdoing. That is likely to include the fact the paper has been substantially written by a large language model (LLM), he argued – which is otherwise difficult to prove given the unreliability of AI detection software.

“If you read these articles, they have a very clear structure that shows AI has done a lot of thinking. The prose is super-polished and verbose, and the tone is very confident, but they are totally empty of meaning. They will always end with a very generic conclusion, such as a call for more research,” he said.

Vainker has urged the British Educational Research Association (BERA) to retract further such papers, containing suspected AI-generated references, following the removal of 19 last month: “These are not edge cases where AI has just been used to improve the language. These phrases are driving the thought and position of a paper, and [it] should be made clear to readers if AI has been used in this way,” he said, noting the requirement of most publishers to disclose the use of AI software.

In a statement, BERA said that its stance on AI, like “that of many other scholarly publishers, is that while generative AI can assist with drafting, summarising, proofreading or refining academic content, it cannot be considered capable of producing an original piece of research or submission without substantial human intellectual contribution and oversight”.

Referencing errors are “not always an indicator that AI has been used in the preparation of a paper”, but the association uses Wiley’s publishing platform, which “automatically screens submissions for referencing errors”, and the statement conceded that “when suspected hallucinated references are identified, a paper does require further scrutiny, particularly when those errors are numerous.

“BERA journal editors are alerted to any concerns for further examination. The outcome then becomes a matter of editorial judgment; depending on the severity of the errors and the context in which they appear, the article may be rejected or the author may be contacted for corrections.”

Source: 

Boris Zhitkov/Getty Images (edited)

The question of whether hallucinated references merit a rejection or retraction, then, is a matter on which intuitions and policies vary widely. But, whatever the threshold for them, should retractions have further career consequences?

Sanders noted that retractions are “not intended as a punishment but as a correction of the literature”. Yet “contaminating the literature with unreliable information” should have further consequences, he believes. Normally, this should entail investigations by authors’ universities, which “have access to documentation that the journal does not…Initial inquiries can be conducted in-house, but once a case proceeds to the investigation stage, to reduce conflicts of interest, the committee should be composed of experts who are not associated with the institutions of the researchers being investigated.”

However, ANRF’s move to require grant applicants to declare any retractions in the past five years, and the reasons for them, suggests that some funders have lost patience with the slow pace of these institutional investigations – and the secrecy that often surrounds them.

The ANRF’s policy shift was lobbied for by Achal Agrawal, founder of the non-profit group India Research Watch, which aims to improve research integrity. He acknowledged the limitations of punishment as a deterrent in a publish-or-perish research culture: “Punishments are always short-term and unsustainable solutions. In an ideal world people should have no incentive to waste their and others’ time publishing low-quality work.”

While panel members can assess each case on its merits, it seems likely that any declared retractions will influence their decisions negatively, and Agrawal acknowledged that this may not always be justified. But until institutions stop “rewarding quantity…these ‘punishments’ are necessary to keep misconduct in check and stop research from descending into chaos, even when sometimes they might be unfair”, he insisted.

Another noted research integrity sleuth, Sholto David, is not so sure, however. As well as the risk of “catching out people in a way that seems unfair, especially to students”, he thinks that putting researchers with retractions on a watch list, or penalising their institutions in rankings, would “seem to risk disincentivising universities from actively retracting their own bad research, and I’m not sure that will have good consequences in the long run”.

According to David, who works at a biotech company in Oxford and who recently received part of a $15 million (£11 million) settlement by a Harvard University cancer research institute after he identified altered images in some of its papers, the responsibility for policing research should remain primarily with “institutions to investigate and sanction individuals, [just as] any other employer should identify and restrict the activities of dishonest employees”.

Yet he conceded that “the current situation seems inadequate” because “people with extensive records of data manipulation are not prevented from contributing further examples of bad science”. Hence, arXiv’s new policy “seems reasonable at first glance”. And if it “turns out to be effective, perhaps it can be implemented more broadly”.

But Miller’s sense is that AI technology is moving much too fast for such a policy to catch on.

“Hallucinated references won’t be a thing in a few years, even months,” he predicted. And it is that speed of advance, he suspects, that is the real reason colleagues are so down on them.

“This outrage is really because academics are terrified that AI can do much of their work better than they can and will soon make them obsolete,” he said, noting that LLMs can “already turn out better papers than many academics”.

“When technology makes your job less valuable then people react negatively,” he said. “The losers in academia will be those not using this technology. They want to use citations to attack and discredit it.”