Systematic reviews are publishing us out of a job

Whenever I ask what a colleague is working on, almost inevitably the answer comes back: “Oh, we’re doing a systematic review on X.”

X seems to be particularly likely to be a topic in health psychology, or psychology in general. I have a critical review and meta-analysis on childhood trauma and paranormal belief in press myself, so I’m as guilty as the next person. But I do wonder whether we are overdoing it.

A systematic review looks at a number of studies in a specific area and uses a protocol to search within certain inclusion and exclusion criteria; it then reviews and appraises each study’s quality. A meta-analysis is a statistical technique that pools the numerical results of several studies to estimate an overall effect and is usually done inside a systematic review. Both can contribute a lot to the literature by giving us an idea of the state of the field.

More importantly, they can inform practice. For example, a 2002 Cochrane review of single-session debriefing, which people routinely used to be given after traumatic events, found “no evidence” that it prevented post-traumatic stress disorder (PTSD) and some signs that it could actually make things worse. The authors concluded that “compulsory debriefing of victims of trauma should cease”, and NICE’s 2005 guideline on PTSD then advised that debriefing “should not be routine practice”.

But if you’re constantly doing systematic reviews on topics that already have them, those reviews will change nothing.

Are there really more of these papers than there used to be, or am I just imagining it? An estimated 29,073 PubMed-indexed systematic reviews were published in 2019 (nearly 80 a day), compared with just 1,432 in 2000. That’s a twentyfold rise. Nor can this be explained by there being more research published. PubMed publications grew 2.6-fold over the same period: eight times slower.

John Ioannidis estimated back in 2016 that more systematic reviews of trials than new randomised trials were published annually. During Covid, 38 systematic reviews of the drug remdesivir were published, a new publication every nine days, yet there were only four major trials to review. And a 2020 editorial in Health Psychology Review admitted that there are often “so many meta-analyses focused on a given phenomenon that it is challenging for any individual to summarise the conclusions”. Its solution was a special issue of meta-reviews.

But meta-reviews can only be useful if the reviewed studies are robust. If not, they offer a textbook case of garbage in, garbage out. Take motivational interviewing. A 2018 review of reviews found 104 reviews, including 39 meta-analyses, yet most of the evidence was rated low or very low quality, and only 11 of 155 comparisons found moderate-quality evidence of benefit.

So why are people reviewing instead of conducting good-quality primary research? The answer is that generating new data is expensive, whether they are quantitative or qualitative. Even recruiting research participants on the Prolific platform needs funding; without it, you’re back to student samples. But funding is getting harder to win. UKRI applications have risen by more than 80 per cent in seven years, while award rates have fallen below 20 per cent – and potentially even lower for the Economic and Social Research Council (which no longer publishes separate figures).

The sub-panel covering nursing and allied health in the 2021 Research Excellence Framework even wondered whether some funders were paying to review old research “rather than the overall development of new knowledge”. The panel noted “a disappointing reliance on iterative systematic reviews that added little to the extant knowledge base”.

But even if the REF doesn’t reward redundant reviews, two additional factors are fuelling the boom. The first is citations. Quality is very difficult to measure, so citations often stand in for it, and they can affect hiring and promotion. But as the psychologist Donald Campbell warned in 1979, the more a number is used to make decisions, the more pressure there is to game it. And since it’s much easier to cite one review than 20 studies, reviews can attract more citations than individual studies. Hence, people want to write them and harvest the citations.

The second is speed. A traditional systematic review took an average of 67.3 weeks to conduct and publish with a team of five, according to a 2017 study. But AI-driven tools, such as Consensus and Rayyan, can now search and screen the literature so much faster. One study, not yet peer-reviewed, reports an AI system reproducing 12 Cochrane reviews in two days.

But while finding and summarising studies is becoming cheap, judging them, critiquing them and building theory isn’t. Back in 2007, an editorial in Annals of Internal Medicine declared that “we urgently need a new type of review”, combining the rigour of systematic reviews with expert context and opinion. So maybe it is OK for me to publish my critical review after all – or at least the critical half. The statistics in this article, for instance, took mere minutes to find with AI (and I did check them) but that left me free for more of the critical evaluation.

But presumably it won’t be long before AI can do the critical stuff too. At that point, we could have one AI-maintained living review per question, updated as new studies appear. Living reviews already exist, and they can work: during Covid, while static remdesivir reviews piled up, Australia’s living guidelines took in each major new trial within a month.

The citation economy will still encourage academics to produce their own systematic reviews, of course. But journals could ask authors to justify why a new review is needed. Only four of those 38 individual remdesivir reviews did so.

The problem is that AI isn’t quite ready yet to conduct systematic reviews. So living reviews still need somebody to keep them alive. Perhaps unsurprisingly, a 2023 survey of 76 living reviews found that many had fallen behind their planned updates.

Until AI is ready, then, we need to fund the human maintenance of living reviews. Even more importantly, we need to better fund and incentivise the generation of new data. Without that, there will be few new studies to review – either by humans or machines. And, ultimately, there will be few academics left to conduct them.

Malcolm B. Schofield is senior lecturer in psychology at the University of Derby.