Enhancing Programming Education Through LLM-Based Learning Analytics and Interventions
C.B.Z. Busropan (TU Delft - Electrical Engineering, Mathematics and Computer Science)
Marcus M. Specht – Mentor (TU Delft - Web Information Systems)
Manuel Valle Torre – Mentor (TU Delft - Web Information Systems)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
The rising number of students in computer science presents a challenge for educators to analyse student work and identify problem areas at scale. Learning analytics systems often fall short in offering detailed, actionable insights that educators can use to enhance their teaching. To address this, we propose a system leveraging Large Language Models to analyse programming submissions and generate actionable analytics. The study focused on two primary research questions regarding the accuracy of LLMs in classifying programming submissions and identifying common issues, and educators' perceptions of the usefulness of the system. Through a system evaluation and focus group, we found that LLMs can analyse SQL assignment submissions with reasonable accuracy and can identify common issues. Educators found the insights potentially useful but noted areas for improvement such as the need for concrete statistics, accurate lists of submissions with specific issues, and more readable reports. Future research should address these implications for the system, evaluate the system in other courses with different programming languages, and involve stakeholders directly related to the course.