What You Show is What You Get!: Gestures for Microtask Crowdsourcing

Crowdsourcing is a valuable tool to gather human input which enables the development of reliable artificial intelligence systems. Microtask platforms like Prolific and Amazon's Mechanical Turk have flourished by creating environments where crowd workers can provide such human input in a diverse and representative manner. Such marketplaces...

HybridEval: A Human-AI Collaborative Approach for Evaluating Design Ideas at Scale

Evaluating design ideas is necessary to predict their success and assess their impact early on in the process. Existing methods rely either on metrics computed by systems that are effective but subject to errors and bias, or experts' ratings, which are accurate but expensive and long to collect. Crowdsourcing offers a compelling way to...

How do you feel?: Measuring User-Perceived Value for Rejecting Machine Decisions in Hate Speech Detection

Hate speech moderation remains a challenging task for social media platforms. Human-AI collaborative systems offer the potential to combine the strengths of humans' reliability and the scalability of machine learning to tackle this issue effectively. While methods for task handover in human-AI collaboration exist that consider the costs of...

Comprehensive viewpoint representations for a deeper understanding of user interactions with debated topics

Research in the area of human information interaction (HII) typically represents viewpoints on debated topics in a binary fashion, as either against or in favor of a given topic (e.g., the feminist movement). This simple taxonomy, however, greatly reduces the latent richness of viewpoints and thereby limits the potential of research and...

Scriptoria: A Crowd-powered Music Transcription System

In this demo we present Scriptoria, an online crowdsourcing system to tackle the complex transcription process of classical orchestral scores. The system’s requirements are based on experts’ feedback from classical orchestra members. The architecture enables an end- to-end transcription process (from PDF to MEI) using a scalable microtask design...

What Is Unclear? Computational Assessment of Task Clarity in Crowdsourcing

Designing tasks clearly to facilitate accurate task completion is a challenging endeavor for requesters on crowdsourcing platforms. Prior research shows that inexperienced requesters fail to write clear and complete task descriptions which directly leads to low quality submissions from workers. By complementing existing works that have aimed...

Improving Reactions to Rejection in Crowdsourcing through Self-Reflection

In popular crowdsourcing marketplaces like Amazon Mechanical Turk, crowd workers complete tasks posted by requesters in return for monetary rewards. Task requesters are solely responsible for deciding whether to accept or reject submitted work. Rejecting work can directly affect the monetary reward of corresponding workers, and indirectly...

External Human-Machine Interfaces: Which of 729 Colors Is Best for Signaling 'Please (Do not) Cross'?

Future automated vehicles may be equipped with external human-machine interfaces (eHMIs) capable of signaling to pedestrians whether or not they can cross the road. There is currently no consensus on the correct colors for eHMIs. Industry and academia have already proposed a variety of eHMI colors, including red and green, as well as colors...

CaptureBias: Supporting Media Scholars with Ambiguity-Aware Bias Representation for News Videos

In this project we explore the presence of ambiguity in textual and visual media and its influence on accurately understanding and<br/>capturing bias in news. We study this topic in the context of supporting<br/>media scholars and social scientists in their media analysis. Our focus<br/>lies on racial and gender bias as well as framing and the...

Characterising and Mitigating Aggregation-Bias in Crowdsourced Toxicity Annotations

Training machine learning (ML) models for natural language processing usually requires large amount of data, often acquired through crowdsourcing. The way this data is collected and aggregated can have an effect on the outputs of the trained model such as ignoring the labels which differ from the majority. In this paper we investigate how label...

A Human in the Loop Approach to Capture Bias and Support Media Scientists in News Video Analysis

Bias is inevitable and inherent in any form of communication. News often appear biased to citizens with dierent political orientations, and understood dierently by news media scholars and the broader public. In this paper we advocate the need for accurate methods for bias identication in video news item, to enable rich analytics capabilities in...

Cyclists’ predictions of what a car driver will do next at intersections

In the Netherlands, 30% of fatal crashes between 2010 and 2015 involved a cyclist [1], with a large portion of these crashes occurring at intersections in urban areas. Contributing factors to driver-cyclist collisions at intersections are not only inadequate visual search, but also incorrect expectations about the other’s intentions [2]....

Utilizing the Potential of the Affected Population and Prevalent Mobile Technology during Disaster Response: Propositions for a Literature Survey

Despite the growing awareness of the untapped potential of the affected population in a disaster situation, their inclusion in a disaster management is extremely limited. This study aims to survey the literature to see whether utilizing the affected people and prevalent mobile technology can be used during disaster response. The idea is to...