Can AI Examine Understanding? Designing and Evaluating ACA-Bot, a Pedagogically Grounded Oral Exam System for Introductory Statistics
MSc Thesis of Julie Bauer, defended on Wednesday, 1st of July, 2026 at DACS/FSE, Maastricht University, The Netherlands
Abstract: Oral exams can capture a student's understanding better than written exams, but scalability and examiner bias remain issues for traditional oral exams. Conversation-based assessment (CBA) with large language models (LLMs) could help address these limitations, but such systems need to be designed carefully to be fair, valid, and reliable. Therefore, this thesis presents ACA-Bot, a speech-based oral examiner that conducts oral exams about statistical topics. The system uses a cascaded architecture with speech recognition, LLM-based dialogue control, and text-to-speech. Its design is guided by predefined pedagogical dimensions, including diagnostic questioning, adaptive questioning, neutrality, content alignment, assessment coherence, prompting consistency, and feedback quality. ACA-Bot was evaluated through a user study, an LLM-as-a-judge evaluation, and qualitative conversation analysis. The results show that ACA-Bot can support the assessment of students’ understanding in some ways. The system performed strongest on pedagogical dimensions that could be controlled through prompts, rubrics, and a predefined topic structure, such as content alignment, neutrality, assessment coherence, and feedback quality. Users also generally perceived ACA-Bot as an assessor, not as a teacher, and found the final feedback useful for identifying strengths and weaknesses. However, ACA-Bot was less consistent in asking diagnostic and adaptive questions. It did not always ask enough follow-up questions and sometimes moved on before the user had fully demonstrated their understanding. The speech modality played an important role in ACA-Bot's limitations. Interruptions and speech-recognition issues influenced whether users had enough opportunity to explain, clarify, or expand their answers. Therefore, ACA-Bot is currently most suitable for formative assessment and exam preparation. Overall, this thesis shows that spoken LLM-based oral assessment has potential, but that pedagogical quality depends strongly on the quality of the spoken interaction.
Cite: Julie Bauer, Can AI Examine Understanding? Designing and Evaluating ACA-Bot, a Pedagogically Grounded Oral Exam System for Introductory Statistics, MSc Thesis, DACS/FSE, Maastricht University, The Netherlands, July 2026.