Code-Aware LLM Prompting in Deductive Qualitative Analysis: A Study in Multi-framework Analysis of Learning Designs
Large Language Models (LLMs) have demonstrated potential for (semi-)automating the qualitative analysis of unstructured data, particularly in deductive qualitative coding using codebooks. While prior research has shown the feasibility of this technology, model performance seems to vary depending on the nature of the constructs being coded. However, existing approaches typically apply a single LLM prompting strategy across entire datasets, often of discourse transcripts or questionnaire data, using a single coding scheme or theoretical frame. This paper introduces an adaptive prompting approach where LLM prompts are customized based on researcher-defined or data-driven rules for specific codes. We apply this approach to analyze the description of 35 multimedia learning designs (comprising 758 items/activities) created by teachers in an inquiry-based learning digital platform. Two human coders and an open-weights LLM (Llama 3.3) coded the dataset attending to three different pedagogical frameworks. Our results indicate that data-driven adaptive prompting outperforms uniform prompting approaches (namely, zero-shot, few-shot with/without context) in terms of agreement with human coders and cost. While the improvement is not statistically significant, the approach offers potential advantages for large datasets (especially, considering the costs), highlighting opportunities for human-AI collaboration in educational data analysis.
2-s2.0-105016166610
Universidad de Valladolid
Universidad de Valladolid
Universidad de Valladolid
Universidad de Valladolid
Universidad de Valladolid
Universidad de Valladolid
École Polytechnique Fédérale de Lausanne
2026
978-3-032-03869-2
978-3-032-03870-8
XXVI, 569
Lecture Notes in Computer Science; 16063 LNCS
1611-3349
0302-9743
427
441
REVIEWED
EPFL
| Event name | Event acronym | Event place | Event date |
Newcastle upon Tyne and Durham, UK | 2025-09-15 - 2025-09-19 | ||
| Funder | Funding(s) | Grant Number | Grant URL |
Regional Government of Castile and Leon | |||
European Union | RYC2021-032273-I | ||
ERDF | PID2023-146692OB-C32,VA176P23 | ||
| Show more | |||