Concurrent Think-aloud Data for Automatic Performance Trust Assessment

Bachelor Thesis (2026)
Author(s)

D.D. Kosecioğlu (TU Delft - Electrical Engineering, Mathematics and Computer Science)

Contributor(s)

M.L. Tielman – Mentor (TU Delft - Electrical Engineering, Mathematics and Computer Science)

C.W. Ning – Mentor (TU Delft - Electrical Engineering, Mathematics and Computer Science)

B.J.W. Dudzik – Graduation committee member (TU Delft - Electrical Engineering, Mathematics and Computer Science)

Faculty
Electrical Engineering, Mathematics and Computer Science
More Info
expand_more
Publication Year
2026
Language
English
Coordinates
52.0027, 4.3752
Graduation Date
23-06-2026
Awarding Institution
Delft University of Technology
Project
CSE3000 Research Project
Programme
Computer Science and Engineering
Faculty
Electrical Engineering, Mathematics and Computer Science
Page Views
105
Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Abstract

As autonomous agents become increasingly integrated into collaborative environments, tracking and calibrating human trust in real time has become more and more crucial to prevent costly outcomes like automation complacency or disuse. However, psychometric tools, such as the Multi-Dimensional Measure of Trust (MDMT), capture only a static retrospective snapshot, missing the moment-to-moment fluctuations of interaction. This study investigates whether concurrent think-aloud (CTA) data can serve as a viable real-time proxy for performance trust. A methodological validation study (N=30) was conducted, using a cooperative game-based grid-world environment where participants worked alongside a teammate programmed to fail in controlled ways across distinct interaction phases. Qualitative protocol analysis successfully mapped transcript indicators to MDMT dimensions and revealed Defiance as a novel behavioral category capturing disuse and takeover actions. Phase-based event analysis showed high temporal sensitivity, with 49.6% of negative performance remarks occurring within a 10-second window following a robot failure. Finally, convergence analysis revealed a striking asymmetry: positive verbal indicators significantly correlated with retrospective MDMT performance trust scores (r=.539,p=.004$), whereas negative verbalizations did not (p=.066), indicating that verbalized frustration captures transient emotional spikes rather than structural drops in stable trust beliefs. These findings demonstrate that while CTA cannot replace post-hoc scales, it offers a powerful, non-invasive complementary tool providing the temporal resolution necessary to understand the dynamic process of performance trust development.

Files

Paper.pdf
(pdf | 0.877 Mb)
License info not available