Evaluation of feedback generated from agent-based social skills training systems
A qualitative analysis on the comprehensibility, usability, and improvement points of the generated feedback
N. Ntasi (TU Delft - Electrical Engineering, Mathematics and Computer Science)
W.P. Brinkman – Mentor (TU Delft - Interactive Intelligence)
Mohammed Al Owayyed – Mentor (TU Delft - Interactive Intelligence)
E. Eisemann – Graduation committee member (TU Delft - Computer Graphics and Visualisation)
More Info
expand_more
The link for the dataset.
https://doi.org/10.4121/85110f4b-40e1-4567-9f6e-e97c6337ad92Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Chatbots are tools that can potentially be utilized in chat-based child helpline training. In this type of training, the quality of the feedback received is of vital importance. This paper aims to analyze the automated feedback generated by such a chatbot. The domains analyzed include user comprehension, usefulness, and potential improvement points. In a user study, a formative assessment and two interviews were conducted for each domain, respectively.
For comprehensibility, all participants could easily understand the feedback report. They found that the bot could be easier to work around after reading the feedback, with the table being of much guidance. They found the transcript to be a welcome addition, but missing constructive feedback. Regarding improvement points, two of them were tightly related to the limitations of the chatbot, rather than the report itself. Extra guidance and instructions were deemed necessary by the participants, alongside an easier-to-read transcript interface.