AI article review ‘better than humans at their best’

Article reviews generated by artificial intelligence are “now better” than the best humans can produce by every measure, two Harvard professors have admitted despite it throwing into doubt the future of their own peer review service.

Gary King, director of the Institute for Quantitative Social Science (IQSS) at Harvard University, and Steven Worthington, director of data science at IQSS, investigated reviews of papers submitted to rapidPeer, their double-blind peer review service for university faculty members.

“When we wrote this, we thought [that] we would find good justification for human reviewing,” King said.

“And therefore, for our rapidPeer organisation…and it’s a beloved organisation,” he said. Since its inception in 2019, rapidPeer has been rolled out to several other institutions.

But that is “not what we found” and subsequently, King said, “we’re going to have to pivot what the organisation does”.

The service, invented by the institute, provides “quality double-blind reviews” of paper drafts before they are formally submitted to a journal. As part of the initiative, reviewers are “paid (well) for their time” and are incentivised to do a “good job”, the paper says. The pair have claimed that editors from major journals have said the reviews are “among the best” they have seen.

The paid-for aspect is atypical in academia because reviewing papers is usually undertaken voluntarily.

The pair took 134 human reviews of 124 scholarly papers submitted between 2019 and 2026 and generated an equivalent AI review using Anthropic’s Claude Opus tool. The papers covered a wide range of research areas including statistical analyses of empirical data and purely qualitative studies.

They chose a different model – OpenAI’s GPT-5.5 – to serve as an AI judge and enlisted three former editors of leading scholarly journals to review 10 randomly selected review pairs and rank them. Both the human and AI judges were given the same rapidPeer criteria to make an assessment about the quality of the review.

AI beat the human review in 100 per cent of the pairs overall when judged by another AI model and 87 per cent when judged by the human journal editors, a paper, which has yet to peer-reviewed itself, says. “The results are unambiguous: No matter how we look at the data, AI wins,” it adds.

Moreover, the AI review had a “higher mean rating” in all seven areas assessed: “sources, weaknesses, methods, theory & idea, framing, communication and figures & tables”.

King said the opportunities for AI to make the review process more efficient are “enormous”. “Imagine a review process that takes a week and a half,” he said.

In peer review, reviewers, authors and editors are people trying not to offend each other, King explained. The AI “takes away that sycophancy”, he said, which he deems a major inefficiency in the current process.

But King, a leading researcher in political methodology who has his own “GaryAI” chatbot on his Harvard profile, said there are also dangers with AI. For instance, “should OpenAI and Anthropic be deciding what’s good science?” he asked.

The paper says that “AI benefits us individually but risks making the collective efforts of science more homogenous and ordinary”, citing studies that show AI tends “toward the conventional, orthodox, and majoritarian ideas”. AI reviews may also have the same issues, “potentially pushing individual papers and science as a whole toward homogenous arguments and conclusions”, the authors warn.

King and Worthington go on to cite potential policy changes that journals could implement given their findings. The most “realistic” one that they cite is that some journals could skip the “routine use of human reviews”.

A journal editor could instead just review a “dossier submitted by the author (consisting of the paper, the AI review generated from the journal’s official AI prompt, and the author’s response to the AI)”. This review could be aided by a new AI review generated by private prompts the editor chooses and, only if deemed necessary, a human review.

Other benefits of AI review that the study identified included greater accuracy and less risk of bias in the process, according to King.

“If we can get the policies right, the job of the editor may then be improved and of more interest to the high-quality people the scientific community attracts for this job,” the authors conclude. 

frances.jones@timeshighereducation.com