The Effects of Crowd Worker Biases in Fact-Checking Tasks

The Effects of Crowd Worker Biases in Fact-Checking Tasks

Draws, T.A. (TU Delft Web Information Systems)
La Barbera, David (Università degli Studi di Udine)
Soprano, Michael (Università degli Studi di Udine)
Roitero, Kevin (Università degli Studi di Udine)
Ceolin, Davide (Centrum Wiskunde & Informatica (CWI))
Checco, Alessandro (Sapienza-University of Rome)
Mizzaro, Stefano (Università degli Studi di Udine)

2022

Due to the increasing amount of information shared online every day, the need for sound and reliable ways of distinguishing between trustworthy and non-trustworthy information is as present as ever. One technique for performing fact-checking at scale is to employ human intelligence in the form of crowd workers. Although earlier work has suggested that crowd workers can reliably identify misinformation, cognitive biases of crowd workers may reduce the quality of truthfulness judgments in this context. We performed a systematic exploratory analysis of publicly available crowdsourced data to identify a set of potential systematic biases that may occur when crowd workers perform fact-checking tasks. Following this exploratory study, we collected a novel data set of crowdsourced truthfulness judgments to validate our hypotheses. Our findings suggest that workers generally overestimate the truthfulness of statements and that different individual characteristics (i.e., their belief in science) and cognitive biases (i.e., the affect heuristic and overconfidence) can affect their annotations. Interestingly, we find that, depending on the general judgment tendencies of workers, their biases may sometimes lead to more accurate judgments.

Bias
Crowdsourcing
Explainability
Misinformation
Truthfulness

http://resolver.tudelft.nl/uuid:1b635898-fe62-42bf-b502-6714953d75b8

https://doi.org/10.1145/3531146.3534629

Association for Computing Machinery (ACM)

2023-07-01

978-1-4503-9352-2

Proceedings of 2022 5th ACM Conference on Fairness, Accountability, and Transparency, FAccT 2022

5th ACM Conference on Fairness, Accountability, and Transparency, FAccT 2022, 2022-06-21 → 2022-06-24, Virtual, Online, Korea, Republic of

ACM International Conference Proceeding Series

Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Institutional Repository

conference paper

© 2022 T.A. Draws, David La Barbera, Michael Soprano, Kevin Roitero, Davide Ceolin, Alessandro Checco, Stefano Mizzaro