Understanding Student Difficulties in Machine Learning Assignments: A Dashboard for Analyzing student-AI interactions

Abstract

Machine learning education presents unique challenges compared to traditional computer science courses: difficulties in actual implementation, differences in background knowledge, and quality of self-study resources. Language models have shown the potential to address these challenges and generate rich student-AI interaction data that may provide valuable insights for learning analytics. However, it remains unclear how such data can be systematically collected, analyzed, and presented to instructors in a meaningful way. To explore that, a case study was conducted in which bachelor students worked on a machine learning assignment using an AI supported programming system JELAI. The collected interactions illustrate how students use AI tools during work. To analyze these interactions, we developed a transformer-based classifier to categorize them into pedagogically relevant question types, and we also compared it with the prompt-based classifier on a classification task. In addition to that, we designed a learning analytics dashboard to visualize categorized interactions and evaluated it through a meeting focus on perceived usefulness. The results indicate that the automatic classification is feasible, but the accuracy is imperfect. The transformer-based classifier showed better performance in a challenging category, while other categories showed similar performance between models. The dashboard was perceived as useful, while also revealing areas for improvement in design and analysis. This thesis highlights the potential and challenge of using student-AI interactions for learning analytics and also motivates future large-scale studies.