The Wizard of Incentive
A Guiding Tool for the Design of Incentive Formulas in Crowdsourcing
V. Macsim (TU Delft - Electrical Engineering, Mathematics and Computer Science)
U.K. Gadiraju – Mentor (TU Delft - Electrical Engineering, Mathematics and Computer Science)
M.L. Tielman – Graduation committee member (TU Delft - Electrical Engineering, Mathematics and Computer Science)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
The rapid growth of artificial intelligence has driven demand for large volumes of real-world data, making crowdsourcing an essential practice. However, crowdsourcing remains largely unregulated, with minimal disclosure of compensation practices in academic literature or dataset documentation. This lack of transparency undermines two important goals: collecting high-quality, realistic data for AI systems, and ensuring fair treatment of workers. Without clear guidance on incentive design, it becomes difficult to distinguish between requesters' lack of knowledge and poor practices—a problem that affects both data quality and worker welfare.
To address this gap, a wizard tool was developed to guide requesters through the process of designing payment schemas for crowdsourcing tasks. A user study was conducted to investigate how structured guidance affects incentive design: first, by comparing designs created with and without the tool, and second, by examining whether the tool produces consistency in compensation decisions across different requesters. The study evaluated both the designs participants created and their feedback on the tool itself.
The analysis reveals three primary insights. First, the tool's primary strength lies in structuring the design process rather than fundamentally altering participants' compensation decisions. The extent to which structured guidance benefited participants depended significantly on their prior experience with crowdsourcing, suggesting that the tool's value is contingent on user expertise. Second, the tool produced convergence around a limited set of high-level design elements, though participants used varied implementation approaches within these patterns, such as specific bonus sums.
These findings indicate that the tool could serve a valuable function in documenting and contextualizing design rationales, capturing the constraints and considerations that shaped dataset creation decisions. However, realizing the tool's full potential as a design aid requires enhancements to customization options and user experience refinement. Despite these limitations, the tool shows promise as an educational resource for introducing beginners to crowdsourcing incentive design, offering a structured entry point into a complex domain.