Effects of Before-Use Explanations on Non-Expert Users' Understanding and Task-Planning Strategies in the Use of Agent Skills
Z. Qiu (TU Delft - Industrial Design Engineering)
Evangelos Niforatos – Graduation committee member (TU Delft - Industrial Design Engineering)
Tianhao He – Mentor (TU Delft - Industrial Design Engineering)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Agent skills provide reusable instruction bundles that guide LLM-based agents through complex workflows. Yet their SKILL.md instructions are written primarily for agent execution rather than non-expert users. Users may know that a skill is available without knowing how to begin, what to provide, or when to adjust its behavior. This thesis asks whether a before-use explanation derived from SKILL.md and reorganized around users' actions can help bridge this gap.
The research followed a two-phase mixed-methods design. In Phase 1, 15 university students used an agent skill and described what they needed before use based on their overall experience. Their difficulties and preferences guided the design of a concise, user-facing explanation derived from SKILL.md. In Phase 2, 24 Master's students took part in a between-subject experiment comparing two pre-task materials: an adapted SKILL.md for the control group and the before-use explanation for the experiment group. Both groups used the same agent skill to complete the same design task. The analysis combined reading behavior, measures of understanding and task planning, interaction logs, and interviews.
The experiment group read the material more completely, reported higher understanding, and demonstrated stronger understanding of available workflow patterns and the pattern used during the task. However, the groups did not differ significantly in task-planning strategies. Qualitative findings suggested that the explanation helped participants recognize input requirements, workflow options, defaults, and checking opportunities, while planning also appeared to reflect prior AI-use routines and task uncertainty. These findings suggest that before-use explanations may help not simply by shortening documentation, but by selecting and reorganizing information that users can act on.
Drawing together the explanation design and empirical findings, this thesis proposes a preliminary translation framework for user-facing agent-skill explanations. The framework highlights three forms of translation: making agent-facing language understandable, connecting procedural knowledge to users’ tasks, and making decision points and opportunities for intervention visible. From a human–AI interaction design perspective, the findings suggest that improving agent-skill use should begin with an information framework that helps users understand the skill and determine how to act, before focusing on interface-level interaction design or visual presentation.