Think-aloud data for automatic trust assessment
RTA x Morality
C. Gheorghe (TU Delft - Electrical Engineering, Mathematics and Computer Science)
C.W. Ning – Mentor (TU Delft - Electrical Engineering, Mathematics and Computer Science)
M.L. Tielman – Mentor (TU Delft - Electrical Engineering, Mathematics and Computer Science)
B.J.W. Dudzik – Graduation committee member (TU Delft - Electrical Engineering, Mathematics and Computer Science)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Trust in an AI teammate is usually measured with a single post-hoc questionnaire, which records a settled judgment but reveals little about when trust shifts or why. Think-aloud protocols are a promising way to recover what questionnaires leave out, because they expose the reasoning behind a judgment rather than only its endpoint, but little work has examined how they reflect trust, and almost none how they reflect its separate dimensions. This paper asks how retrospective think-aloud (RTA) data reflects a user's morality trust in an agent. Using 30 RTA transcripts from a cooperative game with a scripted faulty agent, the study develops a coding scheme for morality-trust markers, links them to the in-game events that elicit them, and compares the result against the morality subscale of the Multi-Dimensional Measure of Trust (MDMT). Morality trust surfaces in RTA narrowly and reactively: mainly as attributions of the agent's intent, strongly skewed negative and ordered in time. Convergence with the MDMT is real but narrow, while the humanising attributions RTA captures have no questionnaire counterpart. The two thus emerge as complementary rather than interchangeable: the questionnaire records a settled judgment, RTA recovers when and why it moved.