Combining LLMs with BDI Systems for Training Children's Helpline Counsellors

Abstract

Child helpline counsellors require various skills and strategies to achieve lasting change in children who require assistance. Typical training methods such as role-play are resource intensive, leading to the development of computer simulation-based training systems where learners counsel the computer which assumes the role of a child requiring assistance. Such systems are limited in their understanding and responses, causing them to appear unrealistic and repetitive. In this paper, we built upon one such rule-based agent through the integration of Large Language Models (LLMs) to vastly expand both the understanding and responses of the agent. We conducted a within-subject experiment with 37 participants who we recruited online through Prolific, where they interacted with both systems, assuming the role of a counsellor. Our results indicate that participants find the integrated system to be human-like in its behaviour, have a more positive attitude towards it, and have a better impression of their overall experience with it. Our thematic analysis revealed that the integrated system felt more adaptive, and engaging, and allowed them to focus more on applying the conversational strategy, while the rule-based system felt scripted and boring. Our work provides an integrated system for effectively training child helpline counsellors and a method by which LLMs and rule-based systems can be integrated in general.