Crowdsourcing as self-fulfilling prophecy

Influence of discarding workers in subjective assessment tasks

Conference paper (2016)

Authors

M.A. Riegler Simula Research Laboratory
Vamsidhar Reddy Gaddam Simula Research Laboratory
Ragnhild Eg Westerdals Oslo School of Arts, Communication and Technology
Pål Halvorsen Simula Research Laboratory
Carsten Griwodz Simula Research Laboratory
Crowdsourcing Multimedia communication Standards Cameras Distortion Quality of service Physiology
More Info
expand_more

Published Date

2016

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Abstract

Crowdsourcing has established itself as a powerful tool for multimedia researchers, and is commonly used to collect human input for various purposes. It is also a fairly widespread practice to control the contributions of users based on the quality of their input. This paper points to the fact that applying this practice in subjective assessment tasks may lead to an undesired, negative outcome. We present a crowdsourcing experiment and a discussion of the ways in which control in crowdsourcing studies can lead to a phenomenon akin to a self-fulfilling prophecy. This paper is intended to trigger discussion and lead to more deeply reflective crowdsourcing practices in the multimedia context.