Crowd-Powered Hybrid Classification Services

None, None; None, None; None, None; None, None; None, None; None, None; None, None

Crowd-Powered Hybrid Classification Services

Calibration is all you need

Conference Paper (2021)

Author(s)

Burcu Sayin (Università degli Studi di Trento)

Evgeny Krivosheev (Università degli Studi di Trento)

Jorge Ramirez (Università degli Studi di Trento)

Fabio Casati (Università degli Studi di Trento)

Ekaterina Taran (TPU)

Veronika Malanina (TPU)

Jie Yang (TU Delft - Electrical Engineering, Mathematics and Computer Science)

Research Group

Web Information Systems

Machine learning Calibration Crowdsourcing

DOI related publication

https://doi.org/10.1109/ICWS53863.2021.00019 Final published version

To reference this document use

https://resolver.tudelft.nl/uuid:c4d35b10-dc9c-45da-be8b-f991bf7b896f

More Info

expand_more

Publication Year

2021

Language

English

Research Group

Web Information Systems

Article number

9590410

Pages (from-to)

42-50

ISBN (print)

978-1-6654-1682-5

ISBN (electronic)

978-1-6654-1681-8

Event

ICWS 2021 (2021-09-05 - 2021-09-10), Virtual at Chicago, United States

Downloads counter

147

Abstract

Hybrid classification services are online services that combine machine learning (ML) and humans - either crowd workers or experts - to achieve a classification objective, from relatively simple ones such as deriving the sentiment of a text to more complex ones such as medical diagnoses. This paper takes the first steps toward a science for hybrid classification services, discussing key concepts, challenges, and architectures, and then focusing on a central aspect, that of ML calibration and how it can be achieved with crowdsourced labels.