Crowdsourcing as self-fulfilling prophecy

None, None; None, None; None, None; None, None; None, None; None, None

Crowdsourcing as self-fulfilling prophecy

Influence of discarding workers in subjective assessment tasks

Conference Paper (2016)

Author(s)

Michael Riegler (Simula Research Laboratory)

Vamsidhar Reddy Gaddam (Simula Research Laboratory)

Martha Larson (TU Delft - Electrical Engineering, Mathematics and Computer Science)

Ragnhild Eg (Westerdals Oslo School of Arts, Communication and Technology)

Pål Halvorsen (Simula Research Laboratory)

Carsten Griwodz (Simula Research Laboratory)

Research Group

Multimedia Computing

Crowdsourcing Multimedia communication Standards Cameras Distortion Quality of service Physiology

DOI related publication

https://doi.org/10.1109/cbmi.2016.7500256 Final published version

To reference this document use

https://resolver.tudelft.nl/uuid:2db223c7-7d59-40d2-8fdd-30e02a56c937

More Info

expand_more

Publication Year

2016

Language

English

Research Group

Multimedia Computing

Pages (from-to)

1-6

ISBN (electronic)

978-1-4673-8695-1

Event

2016 14th International Workshop on Content-Based Multimedia Indexing (2016-06-15 - 2016-06-17), Bucharest, Romania

Downloads counter

167

Abstract

Crowdsourcing has established itself as a powerful tool for multimedia researchers, and is commonly used to collect human input for various purposes. It is also a fairly widespread practice to control the contributions of users based on the quality of their input. This paper points to the fact that applying this practice in subjective assessment tasks may lead to an undesired, negative outcome. We present a crowdsourcing experiment and a discussion of the ways in which control in crowdsourcing studies can lead to a phenomenon akin to a self-fulfilling prophecy. This paper is intended to trigger discussion and lead to more deeply reflective crowdsourcing practices in the multimedia context.