Training machine learning (ML) models for natural language processing usually requires lots of data that is often acquired through crowdsourcing. In crowdsourcing, crowd workers annotate data samples according to one or more properties, such as the sentiment of a sentence, the violence of a video segment, the aesthetics of an image, ... To ensure quality of the annotations, several workers annotate the same sample, and their annotations are combined into one unique label using aggregation techniques such as majority voting. When the property to be annotated by the workers is subjective, the workers’ annotations for one same sample might differ, but all be valid. The way the annotations are aggregated can have an effect on the fairness of the outputs of the trained model. For example only accounting for the majority vote leads to ignoring the workers’ opinions which differ from the majority and consequently being discriminative towards certain workers. Also, ML models are not always designed to account for individual opinions, for simplicity's or performance's sake. Finally, to the best of our knowledge, no method exists to assess the fairness of a ML algorithm predicting a subjective property. In this thesis we address such limitations by seeking an answer to the following research question: how can targeted crowdsourcing be used to increase the fairness of ML algorithms trained for subjective properties' prediction? We investigate how annotation aggregation via majority voting creates a dataset bias towards the majority opinion, and how this dataset bias in combination with the current limits of ML models lead to an algorithmic bias of the ML models trained with this dataset and unfairness in the model’s outputs. We assume that an ML model able to return each annotation of each user is a fair model. We propose a new evaluation method of the ML models' fairness, and a methodology to highlight and mitigate potential unfairness based on the creation of adapted training datasets and ML models. Although our work is applicable to any kind of label aggregation for any data subject to multiple interpretations, we focus on the effects of the bias introduced by majority voting for the task of predicting sentence toxicity. Our results show that the fairness evaluation method that we create enables to identify unfair algorithms and compare algorithmic fairness, and the final fairness metric is usable in the training process of ML models. The experiments on the models point out that we can mitigate the biases resulting from majority voting and increase the fairness towards the minority opinions. This is provided that the workers’ individual information and each of their annotations are taken into account when training adapted models, rather than only relying on the aggregated annotations, and that the dataset is resampled on criteria according to the favoured aspect of fairness. We also highlight that more work needs to be done to develop crowdsourcing methods to collect high-quality annotations of subjective properties, possibly at low-cost.

Human Computation (HC) has established itself to be a powerful tool for carrying out certain simple and repetitive tasks in the form of microtasks, which to this day are still difficult for a machine to automate. With the latest increase in interest in machine learning, HC has similarly gotten more attention as a popular way to acquire training data in large quantities. Traditional microtask crowdsourcing platforms, such as Amazon Mechanical Turk (AMT) or Figure Eight, are typically built using web-based interfaces. However, the speed and quality of data acquired via the crowd are naturally limited by the number of available workers and their skill set. We perceive a grand opportunity in expanding the crowd by exploring alternative means to the traditional microtask crowdsourcing platforms that are reliant on the web-based interface. More specifically, as popular messaging services such as Telegram, WhatsApp and Facebook Messenger are used on a daily basis by millions of people across the world, we propose to perform HC activities inside these services through a text-based conversational agent (or chatbot). We foresee new opportunities arising in conducting HC inside the chatbot, that could leverage the access to a potentially larger and more diverse crowd. In this thesis, we set the first step towards a new alternative to the typical web-based interface used in HC. As a result, we set out to investigate the viability of facilitating microtask crowdsourcing inside chatbots. To this end, we design and implement a chatbot that acts as a medium for the execution of microtask crowdsourcing activities, which is then used for conducting several pilot experiments. In addition, we propose a mapping from Web to Chatbot tasks for several commonly found User Interface (UI) elements inside worker interfaces. Thereafter, we conduct an elaborate experimental campaign to gauge the feasibility and interest of crowd workers to use the chatbot as a new medium for performing generic microtasks. We designed, implemented and executed six common microtask crowdsourcing types; Information Finding, human OCR (CAPTCHA), Sentiment Analysis, Object Labelling, Image Annotation, and Speech Transcription. For each task type, we implemented a microtask in both a web-based and conversational interface. By measuring the execution time, quality of answers and surveying workers’ satisfaction of a total of 316 distinct workers recruited via Figure Eight, we show that chatbots can be effectively used as an alternative to the web-based interface to perform microtask crowd work. We report that out of all workers who participated in the chatbot tasks, 98.3% of the workers indicated a positive experience and were satisfied with their interaction with the chatbot, while performance in terms of task execution time and output quality was in general comparable.