Crowd-Powered Hybrid Classification Services

Calibration is all you need

Conference paper (2021)

Authors

Burcu Sayin Università di Trento
Evgeny Krivosheev Università di Trento
Jorge Ramirez Università di Trento
Fabio Casati Università di Trento
J. Yang Web Information Systems - EEMCS
Research Group
Web Information Systems (EEMCS) (TU Delft)
Machine learning Calibration Crowdsourcing
More Info
expand_more

Published Date

2021

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Electrical Engineering, Mathematics and Computer Science

Department

Software Technology

Research Group

Web Information Systems

Abstract

Hybrid classification services are online services that combine machine learning (ML) and humans - either crowd workers or experts - to achieve a classification objective, from relatively simple ones such as deriving the sentiment of a text to more complex ones such as medical diagnoses. This paper takes the first steps toward a science for hybrid classification services, discussing key concepts, challenges, and architectures, and then focusing on a central aspect, that of ML calibration and how it can be achieved with crowdsourced labels.