Automatic conversational assessment using large language model technology (ICETC 2024)
We presented our first paper about Automated Conversatinal Assessment (ACA) in the International Ccnference on Educational Technology and Computers, ICETC'24, in Porto, Portugal, at the end of September 2024. The prototype and the experient were done, and the paper writen by a talented group of our students in a Master Research Project of our AI program.
Automatic conversational assessment using large language model technology
Jan Bergerhoff, Johannes Bendler, Stefan Stefanov, Enrico Cavinato, Leonard Esser, Tommy Tran, Aki Härmä
Abstract: Student evaluation is an important, yet costly, part of instruction. Traditional exams are a burden for teachers and stressful for students. This paper uses a large language model (LLM) technology to create a system for Automated Conversational Assessment, ACA, where a dialog system, based on content and intended learning outcomes, interviews the student to determine the level of learning. In a pilot experiment in a university course, we found that the ACA system scores correlate with the grades given by a human and also have a positive correlation with the results of a conventional exam of the same students. Based on a questionnaire study, the students responded that the assessment was perceived to be fair and acceptable.