Scaling systematic reviews: how Covidence reduces the research burden
Publishing high quality, highly cited research is a core mission for many higher education institutions. While the rise of digital technologies has significantly reduced the manual workload, it has also introduced new challenges. Which sources can you trust, especially when AI plays a role? And how do you judge whether an AI tool is capable and appropriate, given the risks and expectations for transparency?
Elizabeth Yates, research librarian at Canada’s Brock University, understands the challenges librarians face in providing researcher support, how AI is influencing research quality and how software platforms – such as systematic review management tool Covidence – can alleviate some of the burden faced by librarians.
“Libraries are central to the evidence synthesis research method,” Yates explains. “This methodology relies on comprehensive searching to collect every piece of evidence related to a specific research question. At Brock, we are the experts on campus with the technical skills to design the complex data collection strategies required by researchers.”
Librarians ensure that researchers meet the central tenets of their work: reproducibility, transparency and the reduction of bias. At Brock, a team of librarians led by Yates has undergone specialised training to provide expert support through a tiered service model. Assisting researchers from the very beginning as co-authors, librarians at the university help develop operational research questions, build search strategies and support screening and data extraction.
“There is a saying: ‘garbage in, garbage out’,” Yates says. “If a systematic or scoping review lacks a comprehensive search strategy, it won't collect the evidence needed to answer the research question. This is problematic because evidence synthesis is incredibly impactful. While it originated in healthcare to inform clinical decisions, it has spread to other disciplines to inform programme planning, policy and law.”
Because Brock University doesn’t have a medical school, the research landscape operates differently from clinical institutions. While medical schools focus heavily on systematic reviews to look at healthcare interventions, researchers at Brock are primarily interested in scoping reviews. They want to map the literature on a topic and perform thematic analysis rather than focusing on a narrow clinical question.
“The disciplines we see most often asking for research support are kinesiology, nursing and applied disability studies,” Yates says. “We also have a lot of researchers focusing on other social science programmes. It’s worth noting that, at Brock, many of the librarians are researchers and faculty members themselves. When we partner with a research team, our primary role is to develop the data collection strategy and write the methods section of the manuscript. We are partners from the initial concept through to final publication."
The research pipeline at universities is experiencing disruption from AI, bringing both new opportunities and risks. Emerging red flags around the technology’s use in research include that it represents something of a black box. For instance, unlike a library database, it often isn’t clear how an AI solution interprets search terms. There is also a lack of reproducibility. Results can change, which is a major issue for systematic reviews. Finally, hallucinations cannot yet be ruled out when using AI tools. The fact that AI tools can simply invent subject terms or citations means that any research it is involved with may be unreliable.
“Librarians love new technology. We are usually the first to kick the tyres to see if a tool enhances efficiency,” Yates says. “However, we are also highly analytical. Tools for evidence synthesis must support reproducibility and transparency.”
Despite some concerns, librarians like Yates do see value in AI. “At Brock, we’ve developed an AI position statement,” she says. “We recommend using large language models to break down large topics into manageable chunks or for preliminary search building to generate lists of synonyms. Our general advice is to use AI only within reliable platforms that prioritise accountability.”
The kind of reliable platforms Yates values includes Covidence, which Brock University began using in 2019. The institution’s use of the platform spiked during the pandemic when researchers couldn't enter physical labs and pivoted to evidence synthesis. Boasting faster reviews – an average 35 per cent reduction in time spent per review – expert support and seamless collaboration, Covidence has been hugely beneficial for supporting research at Brock.
“One of Covidence’s standout features is its incorporation of machine learning in the screening process,” explains Yates. “As you screen articles for inclusion or exclusion, the platform learns your behaviour and begins showing the most relevant articles first. While there isn't a strict quantitative figure on the time it's saved us, it has certainly sped up the process significantly, because, by the end, you are easily screening out irrelevant material.”
Crucially, Covidence embraces innovative technologies like AI, but with a critical and deliberate approach. No automation feature is added without thorough evaluation of model performance, human oversight and the potential impact of error. Transparency, reproducibility and trust, supported by openly reported performance metrics, are prioritised over hype.
“The developers of Covidence are very thoughtful. That’s why we trust them over other vendors,” Yates says. “Recently, for example, they worked on a fully automated AI screening tool but chose not to release it because it didn't meet their quality standards. I admire that integrity. In a race to be first to market, they chose to be the best.”
In addition, Covidence understands that research tools are not designed to replace human researchers or librarians. More than just a product, Covidence is a service consisting of training, webinars and 24/7 support from an experienced team of systematic review consultants, removing much of the burden for university librarians like Yates and her team.
“One of the greatest benefits of Covidence is its support for collaboration,” Yates says. “Many library tools are restricted by vendor contracts to internal users only, which creates access issues for external partners. Covidence is different. Once a researcher signs up via our institutional account, they can invite collaborators from any other institution worldwide without barriers. It is a turnkey platform. It’s reliable, the user support is excellent and the knowledge base is extensive. Unlike other library products that may have unintuitive quirks or slow support, Covidence just works.”
