Introductory programming education poses unique challenges; beginners often struggle with abstract concepts, computational procedures, and opaque error messages, often leading to frustration and reliance on memorization.
Large Language Models (LLMs) offer scalable, personali
...
Introductory programming education poses unique challenges; beginners often struggle with abstract concepts, computational procedures, and opaque error messages, often leading to frustration and reliance on memorization.
Large Language Models (LLMs) offer scalable, personalized support but risk hindering independent problem-solving through overreliance.
To explore this tension between support and tension, we investigated the interactions of 18 high school students in JELAI, a programming environment with an LLM chatbot, over a 12-week Python course.
We analysed interaction frequency, the impact of instrumental (adaptive: seeking understanding) versus executive (non-adaptive: seeking solutions) query types on exam performance, and the effectiveness of an intervention promoting instrumental strategies. We also explored the feasibility of automatic query classification using a BERT model, given its potential for scalable analysis of student-LLM interactions.
Higher LLM interaction frequency correlated negatively with exam scores, particularly for students with frequent executive queries.
As expected, executive help-seeking strongly predicted lower grades.
Surprisingly, however, frequent instrumental comprehension queries also correlated negatively with performance, likely indicating that high interaction frequency signals persistent difficulty.
A mid-course intervention successfully reduced the proportion of executive queries and fostered more effective interaction workflows, coinciding with a small average grade increase. Automatic classification successfully distinguished instrumental and executive query types (82.5\% accuracy).
This study highlights that LLM integration requires pedagogical strategies and system designs that actively guide students towards instrumental help-seeking and function as learning partners, rather than mere solution providers.
