Researchers submitting AI-assisted papers they 'don't understand'

An emerging trend of researchers not fully understanding their own AI-assisted research papers is a “serious problem” for academia, according to one journal editor.

Nihar Shah, editor-in-chief of Transactions on Machine Learning Research (TLMR), an open journal on machine learning, has informally investigated the problem in a blogpost published on 16 September, amid a surge in submissions.

According to Shah, the proportion of papers received by his journal that aren’t even sent out for review has climbed from 6 per cent in 2023 to 53 per cent now. And he recently sought to interview 10 authors whose papers were marked for desk rejection to verify their understanding of their papers. Of the seven he was able to interview, three could not answer “basic questions”.

Another three could answer questions about high-level ideas in the paper but “had difficulty when asked further questions on technical details”. The last interviewee answered all his questions – but he “identified a major flaw in that paper”.

Shah writes: “All in all, authors are ultimately responsible for ensuring the accuracy and integrity of the papers they submit under their name. If they cannot explain the basic claims, methods, or technical details of those papers, then this is a serious problem.”

Speaking to Times Higher Education, Shah said he suspected that researchers not fully understanding their papers is “quite a widespread issue across all of sciences”. But “that said, I would also guess it is more [of a problem] in our field due to potentially greater adoption of various AI tools,” he added.

Before Shah published his findings, Thomas Diettrech, editor-in-chief of the physics preprint server arXiv, said on X that he was also seeing a rise in “authors submitting papers whose contents they likely do not understand”.

In answer to his own question about how “the scientific enterprise” should respond, Diettrech mulled ideas such as examining submitting authors’ publication records or credentials in the research area – or, as Shah has done, conduct an oral examination of the author to test their understanding of the paper.

The problem, of course, is that this approach “seems hard to scale”, as Shah put it. But Shah, who is associate professor in machine learning and computer science at Carnegie Mellon University, has been researching how to automate the process.

In a paper yet to be peer-reviewed, Shah and his group have proposed an approach called greCAPTCHA, which uses AI to generate questions intended to assess the authors’ levels of understanding and provides a report on their answers for human review.

Shah and co-authors found that the method “holds broad promise for academic publishing and university pedagogy alike” and said his group was working on a soon-to-be-released portal that will allow users to easily use and customise greCAPTCHA to their particular needs.

“For instance, they may want to change the emphasis on the different types of questions or even consider new types of questions,” Shah said. “Or they may want to check whether the grading rubric aligns with their field’s requirements.” 

TLMR itself is experimenting with a trio of mechanisms “in tandem” to help address the problems presented by AI, Shah said. For instance, in response to what he says is a 13-fold over the past year of single-author submissions, the journal is implementing annual author submission quotas.

Another methods is an “experimental deployment of AI review“ that involves each submission receiving an AI-generated review alongside a human review. And a third involves increasing the emphasis on clear writing in the journal’s acceptance criteria “in terms of what the research has found, why those findings may be of interest to some researchers in the area, and what readers can learn from the work.” In a different blog, Shah says: “If a paper is entirely AI generated with little or no human involvement, then at least with current AI systems, it is unlikely to meet this criterion.”

frances.jones@timeshighereducation.com