Measuring Task Complexity in Human Computation Systems

Abstract

Human computation platforms offer requesters the possibility to outsource human intelligence tasks or HITs to large amounts of workers around the world. Researching the various aspects of HIT design has the potential to improve the human computation experience for both requesters and workers. This thesis researches the relationship between quantitative aspects of HITs and their perceived complexity. Specifically, we consider three categories of task attributes: Metadata, Content and Visual. In order to measure these attributes, we constructed a real-time human computation market crawler that we used to gather HIT data and reinstantiate HITs from the largest human computation market, Amazon Mechanical Turk, and a task attribute analysis pipeline to obtain task attribute measurements from the market data. We present an analysis of the various task attributes, and perform an experiment predicting HIT throughput based on task attribute and market data. Our analysis shows that market statistics, task batch size and semantic content features can be used to predict throughput. We also conduct an initial exploration into measuring subjective HIT complexity as perceived by workers. Based on our analysis and experimental results, we identify three key points of advice for human computation task designers.