Computer Science education, particularly at the beginner level, often presents challenges due to vague and unhelpful error messages. This problem is particularly significant for students with low self-efficacy, leading to hindered learning experiences. Large Language Models (LLMs) offer a promising solution by generating more comprehensible and supportive error messages. This study aims to assess whether the rewriting of error messages using LLMs can improve self-efficacy among high school students, focusing on self-efficacy and study success as indicators of improved learning experiences. Through in-class experiments with 32 participants, the findings revealed that LLM rewritten error messages, although consistent with existing research, did not produce statistically significant effects. Therefore, more research is needed to evaluate their impact on learning outcomes and explore the most effective types of prompt. This research contributes to understanding the role of LLMs in educational settings, providing empirical insights into their effectiveness in real-world scenarios.

Hand gestures play a crucial role in communication, especially in social interactions. This research investigates the viability of using coding schemes to describe hand gestures and how accurately they can be classified in crowded environments by using fine-tuned visual transformers such as VideoMAE. The dataset used during training is based on the Conflab dataset and contains top-view video recordings of social interactions in a crowded social setting. The videos are manually annotated for gesture phases (preparation, hold, stroke, recovery) and gesture units. The two classifiers obtain high accuracies after fine tuning, with an overall accuracy of 95% for the gesture phase classification and 93% for classifying whether a clip is a gesture unit or not. These findings suggest that the proposed approach is effective in crowded environments and can be adapted for real-time applications.

Researching hand gestures in real-world social interactions requires very careful analysis. While gesture coding schemes were created with that purpose in mind, they are not widely utilised in research. Moreover, studies on gesture classification rarely focus on the physical nature of movements involved in gesturing, despite the fact that being able to quantify the motion could reveal useful patterns and correlations. To address those points, this research proposes the following approach: using machine learning models to automatically classify physical features of hand gestures, according to a coding scheme. Two such classifiers were created, for the left and right hand respectively. Overall, the results are quite promising - despite a small and imbalanced training set and complex features both models achieved an accuracy of roughly 60%. Moreover, the results indicate that by avoiding some of the simplifications that this research makes, and by using more balanced training data, the accuracy could be significantly increased. This is concrete evidence that machine learning models can indeed be used to classify the physical aspects of hand gestures, as defined
by a coding scheme, in social interactions in the wild.

With the increasing influence of Machine Learning (ML) on our lives, the need for education on this topic is growing. A key component of education is assessment and improving this aspect could lead to better student learning performance. This study aimed to investigate the influence of different assessment methods on students' learning performance in k-means clustering. Two different assessment methods were used: a closed-book problem-based assignment and an open-book short answer exam. Participants were notified of their assessment method, after which they were instructed to watch a video lecture and take the assessment. Results show a significantly improved learning gain when using the open-book assessment, where learning gain was defined as the difference in score between the pre- and post-test. Between these two methods the open-book assessment is therefore favourable. However, future research is needed to develop a validated concept inventory for k-means clustering and identify other possible assessment techniques.

Machine learning education often involves complex topics that can be challenging to teach engagingly, leading to difficulties in maintaining student focus and achieving optimal learning outcomes. This study aims to bridge the gap between machine learning-specific teaching techniques and those centred on student engagement by conducting a comprehensive analysis of related works and an empirical experiment. The related works section reveals differences between traditional and engagement-focused teaching methods. To address the knowledge gap regarding the impact of engagement-focused methods on learning outcomes, a controlled experiment was conducted, comparing a conventional 16-minute video lecture followed by practice questions against the same content divided into four shorter video segments, each followed by a subset of the practice questions. The results demonstrate that the experimental group achieved significantly higher average quiz scores and reported consistently higher satisfaction ratings, suggesting that even a simple engagement-boosting technique can substantially improve learning outcomes and student satisfaction in machine learning education. This study highlights the importance of prioritising student engagement as the field of machine learning continues to evolve.

This study investigates the impact of scaffolded assignments on student learning, confidence, and the development of an empirical mindset in a Machine Learning (ML) course at TU Delft. Unlike traditional Computer Science subjects, ML requires an experimental approach, challenging students used to design-first methodologies. Through surveys of 25 students from the CSE2510 course, the study found that scaffolded assignments significantly enhance student confidence and perceived learning benefits, despite no positive correlation between the number of assignments completed and course grades. Qualitative feedback highlighted the value of scaffolded assignments in understanding the ML design process by providing structured guidance and enabling practice of specific sub-tasks. These findings suggest scaffolded learning is crucial in developing an experimental mindset and boosting student confidence in ML education. This study also proposes a second methodology for future research during an edition of the course to further explore this topic.

The rapid advance of AI and ML asks for better and earlier education on these topics. However, research on teaching AI and ML topics is relatively underdeveloped. Especially applying the teaching method gamification has not yet been thoroughly tested. This research aims to explore the effectiveness of gamifying ML education in high schools. An experiment was designed based on previous research. This experiment revolves around a computer game where students learn to train a car to drive itself. Despite small sample size, results show increased motivation and low anxiety.

The increasing demand for machine learning expertise calls for effective teaching methods for university-level courses. This research compares static versus interactive teaching methods in the context of machine learning, with the latter focusing on the student engaging more with the material. Specifically, this study investigates the impact of interactive visualisations on students' understanding of receiver operating characteristics (ROC) curve analysis, a critical concept in evaluating machine learning algorithms. Traditional static teaching methods often fall short of conveying complex ideas like ROC curves, which are pivotal in various fields, including medicine and psychology. This research also compares the efficacy of interactive versus static visualisations in enhancing student motivation. Twenty first-year computer science students from Delft University of Technology participated in the experiment and were randomly assigned to control (static visualisation) and experimental (interactive visualisation) groups. The results of the experiment were determined by analyzing the pre- and post-test scores, along with surveys measuring motivation. These indicate significant improvements in understanding for both groups, with a greater gain observed in the experimental group. This suggests that interactive visualisations may offer a superior approach to teaching complex machine learning concepts, but the experiment conducted in the study does not show a statistically significant difference between the static and interactive visualisations. The research also compares the student's motivation after completing an instructional material focused on the ROC, but the interactive visualisations did not provide better results. The study underscores the potential of interactive teaching tools to enhance educational outcomes in machine learning and highlights the need for further research into interactive methods for the teaching of machine learning.

With the increasing development of artificial intelligence (AI), there is a more significant opportunity for humans and agents to collaborate in teamwork. In Human-Agent Teamwork (HAT) settings, collaboration requires communication, and the agent displaying emotion can impact how human teammates communicate and work together with the agent. This study investigated the impact of an explainable agent expressing emotion within explanations in a teamwork setting. We investigated how integrating an emotional component into an agent’s explanations influences trust in the agent, as well as humans’ perceptions of the agent’s anthropomorphism, animacy, likeability, and overall team performance when collaborating with the agent. With this goal, a pre-study was conducted using a focus-group meeting to investigate the relevant emotions to display in a simulated Search and Rescue (SAR) task and how these emotions can be incorporated into Explainable AI (XAI). Next, we conducted an in-between subject controlled experiment to study the effects of emotional components in explanations. The participants were divided into experimental and control groups, collaborating with agents that either displayed emotion or no emotion. The participants had to carry out a SAR task where they worked together with the agent to rescue victims. Our results confirmed that an agent displaying emotions increased perceived likeability, animacy, and anthropomorphism. Among these three, likeability and animacy are positively associated with trust. In contrast, an increase in anthropomorphism is associated with a decrease in trust. From the results, we could not conclude that team performance is directly affected by having emotion in the explanation. However, the results showed that emotion increases the messages sent from the human to the agent, and this increase in communication led to higher team performance.

Human-agent teamwork (HAT) is becoming increasingly prevalent in fields such as search and rescue (SAR), where effective collaboration between humans and artificial agents is crucial. Previous studies have shown that trust plays a pivotal role in the success of HATs, influencing decision-making, communication, and potentially overall team performance.

This research investigates the impact of agent-provided explanations about the agent's trust in humans (artificial trust) and corresponding behavior changes on human trust in the agent and their satisfaction with explanations during a simulated SAR task. Two types of explanations were explored: Trust-Explained (TE) explanations, where the agent explains its trust level and trust-based decisions, and Trust-Unexplained (TU) explanations, which solely describe the agent’s behavior without reference to trust dynamics. Besides, this research also investigates the correlation between human trust and explanation satisfaction, and in the end, whether the differences in the provided explanations result in differences in team performance and artificial trust.

The study involved 40 participants divided into two groups: an experimental group (the trust-enhanced explanation group) receiving TE explanations and a control group (the non-trust explanation group) receiving TU explanations. Participants' trust in the agent, satisfaction with the explanations, and team performance and artificial trust were measured and analyzed. Contrary to initial expectations, no statistically significant differences in explanation satisfaction and human trust in the agent were found between the two groups. However, a strong positive correlation was observed between participants' satisfaction with the explanations and their trust in the agent, indicating that explanation quality plays a crucial role in human trust development. Furthermore, no significant differences in team performance were detected, suggesting that trust explanations may not directly influence task outcomes. In the analysis of artificial trust, the agent in the trust-enhanced explanation group exhibited more conservative adjustments in trust levels compared to the non-trust explanation group. This conservative approach may have influenced players in the trust-enhanced explanation group to adopt a more cautious or deliberate decision-making process, potentially prioritizing the comprehension of explanations over the optimization of task performance.

For future research, it may be worth delving deeper into the influence of trust explanations on user behavior, the more complex HAT task environments, the relationship between artificial trust and user behavior, the dynamic and adaptive explanations, and the causal relationship between explanation satisfaction and human trust in the agent to understand further how trust can be fostered in HAT.

Communication is one of the main challenges in Human-Agent Teams (HATs). An important aspect of communication in HATs is the use of explanation styles. This thesis examines the influence of an explainable agent adapting its explanation style to a supervising human team leader on team performance, trust, situation awareness, collaborative fluency, explanation satisfaction, understandability, and user-awareness. To perform a simulated Search And Rescue (SAR) task, a HAT is designed. With this design, a pre-study is then conducted using a questionnaire to discover the best-ranked explanation styles in the most important situations of the SAR task. Next, the user-study is carried out with 46 participants, using the HAT design and analysed data from the pre-study. There are two conditions: the agent adapting the explanation style to the human team leader and the baseline condition where the explanation style is randomised. The results show that the subjective measurements of trust, understandability, explanation satisfaction, and perceived user-awareness are significantly higher in the adaptive agent condition group. The same cannot be concluded for objective measurements such as team performance and situation awareness.

Programming robots with verbal commands is limited by the capabilities of the utilized natural language parser. A simple natural language parser which can understand only keywords and small phrases may be easy to use, but limited in what it can interpret and convey. Alternatively, a natural language parser which understands more complex commands can be used to convey more nuance, but can be more difficult to use and create. It is unclear when having complex verbal commands available is preferable to having only simple verbal commands available. Here we show that using natural language parsers which understand more complex commands are preferable when teaching a robot, both in terms of user preference and objective metrics such as completion time and accuracy, but only when the task is complex as well. During a preliminary wizard of oz experiment, we observed what types of phrases users use to correct the robot during a pose imitation learning task, in order to create multiple natural language parsers which allowed for different levels of complexity in given verbal feedback. In a follow up experiment, in which 24 users utilized these parsers in a similar task, the users reported finding the more complex ones to be more useful and satisfying to use. Additionally, the more complex parsers also led to a higher objective similarity between the pose that the user wanted to convey and the final attained pose by the robot. However, this last result was only found for poses which required a comparably high effort on part of the user.

STUDY OBJECTIVE: Suicide crisis chat counsellors work in an environment which de- mands high emotional and cognitive awareness. A shared opinion among counsellors is that as the chat conversation turns more difficult it takes longer and more effort to come up with a response. Supportive technology might resolve this ”writer’s block” by giving inspiration in the form of responses other counsellors gave to similar situations. 113 Suicide Prevention has an extensive chat corpus of previous conversation between help-seekers and counsellors that can be leveraged to find these situations. This thesis focuses on the design, development and experimental evaluation of a support system that aims to solve the problem of writer’s block for counsellors in suicide crisis chats.
METHODS: A context-based retrieval system was build. SIF sentence embedding was used to find similar partial chats using cosine similarity. The chats in the corpus were split up and embedded using a sliding window approach, to retrieve only the relevant parts in the chat and reducing reading time for the counsellor. A within-subject experimental design (n=24) with three conditions (no support, support system, expert advice) was used to measure the system’s usefulness and to test the hypothesis that the system has a noticeable difference on counsellor responses, by using experts to blindly label the responses. Furthermore, another within-subject experimental design was used to test the hypothesis that the counsellors can distinguish be- tween partial chats provided by the retrieval system and partial chats randomly selected from the corpus. For this part of the evaluation, counsellors blindly rated suggestions on how much they related to the context of a given chat.
RESULTS: Counsellors rated the support system with a 71.04 on the SUS questionnaire, cor- responding to a good adjective rating. There was variability between chats for counsellor rating for utility of the information provided by the support system. Experts noticed a significant difference between the support and expert advice conditions. Lastly, the suggestions based on the context positively affected counsellor rating of relatedness to context of the chat. CONCLUSION: Technology using NLP techniques can provide useful information for only chat counsellors to help with writer’s block. Improving the quality of the recommendations, is expected to help improve the usefulness of the system.

Objective: This thesis aided the 113 Suicide Prevention (113), the national suicide prevention center for The Netherlands, by investigating technical solutions for the helpline, implementing an e-learning prototype housing six suicidal personas within a conversational agent model, and evaluating and analyzing an experiment on its effect, which entailed interactions with one, two, and three simultaneous chats. Methods: The thesis conducted a participant observation with a total of seven triage-psychologists, organized three focus groups including triage-psychologists, managers, and training personnel with nearly forty participants, and administered an evaluation with thirty participants that included six triage-psychologist and twenty-four counselors regarding a prototype to assist in the training of 113's triage-psychologists. Prototype: The system specification provided a prototype with six personas where triage-psychologists can practice against one or many chatbots, or conversational agents, in different situations that pertain to training for 113. The conversational agents design was based upon the Rose of Leary interpersonal stance and the Beliefs, Desires, and Intentions (BDI) design paradigm. The system focused on how a conversational agent must react to triage-psychologists' inputs with respect to the subtleties in interpersonal communication and negotiation as it pertains to the 113 suicide helpline. Results: Evaluation results indicate that triage-psychologists found the learning environment motivational and the events in the environment as socially realistic. With the additional number of chats, counselors experienced an increase in three measurable areas: 1.) mental effort; 2.) situational awareness demand; and 3.) situational awareness supply; even so, counselors were positive about all learning aspects regarding the new software environment. Conclusion: This work identified the natural language processing, the BDI reasoning model plus natural language generation, and the usability and quality of the prototype as three areas of focus for 113 as they continue to improve their management of the helpline, its training, and research on suicide.

Making virtual characters seem sentient is the main goal of the study of believable virtual characters. Unbelievable behaviour, especially eye gaze behaviour, could make the whole virtual character seem less believable. In human-human social interactions eye gaze is used to interact non-verbally. People also apply Theory of Mind (ToM) to reason about other people's mental states. If a virtual character were to respond to a person's gazing behaviour with its own gazing behaviour, will it seem more believable and socially present to this person? Do people look at- and follow the gaze of such virtual characters more? Do people apply ToM to reason about the mental states of such virtual characters? This study employed a within-subject design (n=24) with two conditions that were given at the same moment. The conditions were represented by two virtual characters where one had gaze aware behaviour and the other did not. Using a think-aloud procedure during and questionnaires after the Virtual Reality experience the believability, social presence and the application of ToM were assessed. Gaze behaviour with respect to the virtual characters was assessed as well using the onboard eye-tracker of the Virtual Reality Head Mounted Display. The results suggest that virtual characters were more believable and socially present when they respond to people's gaze behaviour through their own gaze behaviour compared to a virtual character without such gaze behaviour. Results also showed that in some situations people looked more at the virtual character without gaze aware behaviour than at the virtual character with such behaviour. However, gaze following behaviour and whether or not people apply ToM were inconclusive. In conclusion, virtual characters that respond to people's gaze behaviour through their own gaze behaviour are perceived as more believable and more socially present than virtual characters without such gaze aware behaviour.

This thesis project worked towards extending the interactive capabilities of the RoboTutor Nao, by enabling it to answers natural language questions about topics such as computers, robots and the Nao itself. The main questions we focused on were 1) what a system for answering questions in Dutch and intended for elementary school age children should look like, 2) how the interaction with users should be structured for a smooth interaction and 3) what type of experimental studies and metrics are best for evaluating the quality of the QA system and assess its impact on users. We chose an existing English-language QA system and adapted it to work with Dutch content. An interaction manager was developed for handling the interaction between the system and users. Finally, the system was configured so that it could be interacted with via the Nao robot. The system was then evaluated via two experimental studies. First a quantitative evaluation without users was carried out, evaluating the performance of the system using a collection of test questions. Secondly, a field study with users was done. We took the robot to a day care center where children could ask it questions and assessed the effects of these interactive Q\&A sessions. Results from the quantitative evaluation highlighted slow answering times as the main shortcoming, while recall (70%) and precision (50%) scores were more competitive. The field study showed that even though the occasional repetitiveness and lack of speed were picked up on by users, this did little to curb their enthusiasm. Interaction was smooth, users were interested in many topics the system supported and they continued interacting with it despite the occasional mishap. Overall, the results indicate that, with some speedup, the delivered QA system would be a suitable choice for answering questions about the mentioned topics.