AI won’t replace qualitative researchers – it might help them

Whether you’re excited or sceptical about the use of AI in research, you may be curious: can AI help researchers to conduct qualitative analyses? How are these tools best used? What are the risks? And how might AI be used to support researchers, rather than replace them?

We compared the findings of large language models (LLMs) with the analytic process of a human researcher. In the end, the LLM outputs were almost identical to the human analysis, but here’s what excited us the most: the LLMs didn’t just replicate our work, they offered insights that we hadn’t spotted, challenged our initial assumptions and prompted us to reflect further on our own interpretations. 

• Balance human intuition with machine efficiency in scientific research
• Spotlight guide: GenAI as a research assistant
• Ethical AI: safeguards for the research revolution

Our team, partnered with Ipsos UK, analysed 138 short stories written by young people aged 13 to 25 in and around Southampton. The stories focused on the relationship between identity, food choices and social media in adolescents. It was rich, complex data that would usually take considerable time for humans to analyse and interpret. We estimated that the GPT-o1 analysis took about 12 hours. It took the human researchers about 64 hours (across 16 weeks) to conduct the full narrative analysis of all 138 stories, while documenting findings throughout the process. Using a narrative analysis framework, the stories were analysed:

• by a human qualitative researcher
• using Anthropic’s Claude 3 Opus
• using OpenAI’s GPT-o1.

How we conducted the analysis

We then compared the three sets of findings to see how closely aligned they were, and where the LLM “interpretations” differed from those of the human researcher. To guide the analysis, we developed a four-step framework:

Step 1: Develop a structured analysis plan

We established clear roles for human and LLM analysts from the outset: the human analysis was conducted first, followed by the LLM analyses. We treated each LLM as a member of our research team – not as an all-knowing oracle but as a collaborator with strengths and limitations. 

Step 2: Select the best model for the task

We chose the LLMs that were best suited to our analytical tasks, taking into account the models’ context window, speed, and analytic and reasoning capabilities. Both models were the most advanced LLMs available at the time.

Step 3: Format data appropriately

LLMs respond best to clearly structured text. We created a consistent format for inputting all our data into the LLMs using JSON (JavaScript object notation).

Step 4: Use prompt engineering and optimisation

We prompted the LLMs to ask us questions about our project and refine our instructions, so that they could generate the most effective prompts for us to use for analysis. This wasn’t about handing everything over to a machine. It was about finding the right balance between human subjectivity and computational efficiency.

Comparing the human and LLM ‘decision-making’ processes

The LLMs conducted the narrative analysis incredibly quickly. Claude 3 Opus and GPT-o1 provided us with a detailed explanation of their reasoning for analysing each story in under a minute. 

We already know that LLMs are fast workers, but also that their outputs can be inaccurate. So, how did we check that the models had conducted thorough, credible analyses? 

We asked each model to provide a detailed account of their “decision-making” process, including how and why they categorised stories into narrative groups. We also asked them to justify their interpretations and provide supporting illustrative quotes from the stories.

This process quickly exposed our first major challenge: the models’ context window. Asking the LLMs to process and group all 138 stories in one go was too much, as it would be for a human. This led to some stories not being grouped at all and others being assigned to the wrong groups. To tackle this, we made two crucial changes:

• We reformatted our data from Markdown into JSON because JSON uses explicit “key value” pairs, which allowed the model to understand the relationship between the story text, the participant ID and their age, something Markdown is not designed to do.
• We provided the LLMs with each story one at a time. This “chunking” of the data massively improved accuracy and eliminated the errors.

But the benefits went far beyond simply achieving accuracy. The process transformed the LLMs from a simple analysis tool into a collaborative partner. Their additional insights identifed narrative groupings that challenged our assumptions, prompting us to think more about our interpretations. 

Qualitative researchers may ask: “What about reflexivity and subjectivity, the distinctly human elements of qualitative analyses?” We believe that LLMs have the potential to support researchers to think reflexively, and that by providing alternative interpretations, LLMs can help us to consider the range of biases and assumptions that underpin the analyses we conduct.

Top tips for using LLMs in your qualitative analysis

• Stay sceptical. Always review LLM-generated content for accuracy and rigour. Interrogate LLMs if you’re unsure about the origins of their “interpretations” and be on the lookout for hallucinations.
• Proceed responsibly. Data security is critical when using LLMs. Researchers must be extra careful that their analysis strategy complies with GDPR and institutional ethics requirements.
• Be transparent. Keep a detailed record of how you use LLMs in your work. When publishing or presenting findings, be prepared to explain your process in detail.
• Think collaboratively. View LLMs as tools to strengthen your research. The question is not: “How will AI replace us?” but rather “How can we work with AI safely and effectively?”
• Be mindful. We haven’t touched on the ethical issues associated with AI…but it’s important to keep these in mind. AI has the potential to bring huge benefits to society, but it also poses a threat to our environment and to vulnerable communities who are exploited so we can access well-trained, safe LLMs. 

Ultimately, our work suggests that when used carefully and responsibly, LLMs have the potential to be valuable qualitative research collaborators, helping us to maximise the impact of our research for the people and communities we wish to help. 

Sarah Jenner is a lecturer in child and adolescent health at the University of Southampton. Dimitris Raidos is an associate director at Ipsos UK.

If you would like advice and insight from academics and university staff delivered direct to your inbox each week, sign up for the Campus newsletter.