Retrospective Think-Aloud Data as a Window into Performance Trust
A Human-Agent Collaborative Game Study
A.E. Todorov (TU Delft - Electrical Engineering, Mathematics and Computer Science)
M.L. Tielman – Mentor (TU Delft - Electrical Engineering, Mathematics and Computer Science)
B.J.W. Dudzik – Graduation committee member (TU Delft - Electrical Engineering, Mathematics and Computer Science)
C.W. Ning – Mentor (TU Delft - Electrical Engineering, Mathematics and Computer Science)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Trust between users and autonomous agents is dynamic, yet most measurement instruments capture only a static end-state. Think-aloud protocol offers a window into the reasoning behind trust judgements, but no prior work has examined how retrospective think-aloud (RTA) data relates to performance trust. This paper presents a user study in which 30 participants interacted with a scripted AI agent in a collaborative box-moving game, verbalised their thoughts while watching a recording of their own gameplay (RTA), and then completed the Multidimensional Measure of Trust (MDMT). A deductive content analysis grounded in the MDMT Reliable and Capable subscales produced a 16-code scheme applied to all 30 transcripts. Negative trust language in RTA tracked the game's scripted trust trajectory, peaking during the most severe failure phase, but did not relate to participants' final MDMT scores - suggesting RTA's diagnostic value lies in the timing of this language across the session, rather than its overall volume. These findings support RTA as a method for capturing within-session trust dynamics that static questionnaires cannot surface.