SceneFinder: A collaborative game that collects tacit knowledge from crowds

Abstract

The manual process of collecting and labelling data required for machine learning tasks is labour-intensive, expensive, and time consuming. In the past, efforts have been made to crowdsource this data by either offering people monetary incentives, or by using a gamified approach where users contribute to databases as a side-effect of playing an enjoyable game. However, most of these efforts focus on using a competitive setting to incentivize players. This sometimes results in users spamming the dataset for personal gains. Research is lacking in how a collaborative setup, where players work together to make decisions by consensus, can be used to source knowledge that is more accurate and reliable. This paper describes the design and evaluation of SceneFinder, a game that aims to crowdsource reliable and diverse textual data about scenes (such as rooms, parks, monuments, etc) and the tacit knowledge relevant to them, such as information about their contents, their purpose and their surroundings. SceneFinder makes use of a collaborative setup that elicits a relevance based ranking of facts about these scenes, that distinguishes it from existing games in the field.