HybridEval: A Human-AI Collaborative Approach for Evaluating Design Ideas at Scale

Evaluating design ideas is necessary to predict their success and assess their impact early on in the process. Existing methods rely either on metrics computed by systems that are effective but subject to errors and bias, or experts' ratings, which are accurate but expensive and long to collect. Crowdsourcing offers a compelling way to...

Editorial: Human-centered AI: Crowd computing

Scriptoria: A Crowd-powered Music Transcription System

In this demo we present Scriptoria, an online crowdsourcing system to tackle the complex transcription process of classical orchestral scores. The system’s requirements are based on experts’ feedback from classical orchestra members. The architecture enables an end- to-end transcription process (from PDF to MEI) using a scalable microtask design...

Using Worker Avatars to Improve Microtask Crowdsourcing

The future of crowd work has been identified to depend on worker satisfaction, but we lack a thorough understanding of how worker satisfaction can be increased in microtask crowdsourcing. Prior work has shown that one solution is to build tasks that are engaging. To facilitate engagement, two methods that have received attention in recent HCI...

CaptureBias: Supporting Media Scholars with Ambiguity-Aware Bias Representation for News Videos

In this project we explore the presence of ambiguity in textual and visual media and its influence on accurately understanding and<br/>capturing bias in news. We study this topic in the context of supporting<br/>media scholars and social scientists in their media analysis. Our focus<br/>lies on racial and gender bias as well as framing and the...

Semantic Web and Human Computation: The status of an emerging field

Characterising and Mitigating Aggregation-Bias in Crowdsourced Toxicity Annotations

Training machine learning (ML) models for natural language processing usually requires large amount of data, often acquired through crowdsourcing. The way this data is collected and aggregated can have an effect on the outputs of the trained model such as ignoring the labels which differ from the majority. In this paper we investigate how label...

A Human in the Loop Approach to Capture Bias and Support Media Scientists in News Video Analysis

Bias is inevitable and inherent in any form of communication. News often appear biased to citizens with dierent political orientations, and understood dierently by news media scholars and the broader public. In this paper we advocate the need for accurate methods for bias identication in video news item, to enable rich analytics capabilities in...