Machine learning (ML) systems for computer vision applications are widely deployed in decision-making contexts, including high-stakes domains such as autonomous driving and medical diagnosis. While largely accelerating the decision-making process, those systems have been found to suffer from a severe issue of reliability, i.e., they can easily fail on serving data that are slightly different from the data captured during their training phase. Such an issue has resulted in undesired outcomes with safety, ethical, and societal concerns across various applications, such as numerous examples of semi-automatic cars causing accidents on the road.
In this thesis, we hence develop a system in order to support ML practitioners in debugging their computer vision models, even before deploying them and having access to serving data.

We take inspiration from prior ongoing works in order to formulate the current diagnosis problem, identify its challenges, and envision a human-computation-based solution. We then thoroughly analyse the requirements for developing a system instantiating the solution, actually design such a system, and implement it in a well-functioning, full-fledged, highly-modular, and easily-customizable system.
The solution is based on the definition of human computation operations, that, altogether, allow to a) identify the mechanisms a human would expect the model to learn in an ideal world, b) identify the mechanisms the model has actually learned (via annotations of saliency maps), and c) to compare these two sets of mechanisms to conclude about the good behavior of the model. The solution is especially made to account for certainty issues in the work of the human workers, and to handle ambiguous granularities in the concepts the model might have learned.
To the best of our knowledge, our work is the first system that allows an ML practitioner to first identify their own goals for debugging a model (among a large diversity of goals), accounting for their limited monetary budget, then to configure a debugging session according to these goals, and finally to fully-automatically run the system with such configuration to obtain a model debugging report.

Finally, we conduct a thorough investigation of our system. First, we set-up to understand the correctness and informativeness of the outputs, by running the system with various configurations on different models trained using various datasets, for which the biases are more or less controlled. This first evaluation particularly shows that the outputs and its implementation are correct. With these outputs, we are able to identify the biases that have been injected in the model, as well as to learn about previously unknown behaviors of highly-common models that are used by many practitioners.
Second, we evaluate the cost-effectiveness of running the system. For that, we ran tests in two settings: when the human workers might make mistakes (e.g., due to a lack of expertise, the complexity of the task, or inattention), and when human workers are fully accurate. We vary the configurations of the system (e.g., the order in which the human operations are conducted, the number of workers allocated at the start of the debugging session) within the two settings, and we observe how the number of human operations needed evolve, in order to reach correct system outputs. We find that the system's output is potentially relevant, informative and complete. The system output provide an in depth analysis of the model's behaviour and unravel what the model comprehends, where it falls short, and what it should ideally have grasped.

All in all, in this thesis, we build the system and thoroughly evaluate it. While we identify a number of conceptual and practical limitations of this system (e.g., difficulty to annotate concepts, potentially high cost), our work constitutes a first step towards developing complete solutions to help practitioners debug their system. We encourage readers to build on our work, in order to further optimize our system for cost. Note that we make all our code publicly available for anyone to re-use our system, or reproduce our experiments. ... Read more

Machine learning is still one of the most rapidly growing fields, and is used in a variety of different sectors such as education, healthcare, financial modeling etc(Jordan and Mitchell 2015). However, along with this demand for machine learning algorithms, there comes a need for ensuring that these algorithms are fair and contain little to no bias. Tools like Fairlearn and AI Fairness 360(AIF360) allows developers and data scientists to examine their codebase according to specified fairness metrics and mitigate any fairness related issues. This study aims to determine how practitioners use the separate toolkits and whether their practices are differ by the toolkit they choose to use. To do this, we conducted 29 think-aloud interviews with industry practitioners to understand how they would use Fairlearn and AIF360 in practice. The results show that fairness is a socio-technical challenge. While the toolkit does allow for participants to be structured in their approach, and raise awareness for fairness related harms, at the end of the day the fairness toolkit only provides technical help to find harms that the individual was already aware about. Based on the findings, we then suggest the design for a fairness toolkit that can help practitioners approach fairness in the most ideal manner. This toolkit would include a way to have interdisciplinary collaboration, have a larger focus on explainability, and give clear guidance to its users regarding fairness related harms. ... Read more

To encourage ethical thinking in Machine Learning (ML) development, fairness researchers have created tools to assess and mitigate unfair outcomes. However, despite their efforts, algorithmic harms go beyond what the toolkits currently allow to measure. Through 30 semi-structured interviews, we investigated whether data scientists are constrained to only thinking about issues that can be tackled with these toolkits when using them in practice. The results of a comparative assessment of approaches with and without a toolkit indicate that although they can be incredibly effective, toolkits shouldn't replace educating on sources of harm and can even have hazardous consequences when improperly used. We discovered that while fairness toolkits increase practitioners' awareness of several specific sources of harm, such as questionable attributes or data sampling techniques, their greater power lies in fostering discussions about ML systems' propensity to treat individuals unfairly. On the contrary, we observed that these toolkits do not significantly help in the data documentation process, and, from observing our study participants, we also infer a risk of them blindly evaluating and optimizing for undesired outcomes as a result of choosing metrics and mitigations on unfounded or incomplete assumptions. This work supports future improvement of toolkits by providing a breakdown of perspectives around various sources of harm and reasoning about the ones that get frequently overlooked. ... Read more

The increasing dangers of unfairness in machine learning (ML) are becoming a frequent subject of discussion, both, in academia and popular media. Recent literature focused on introducing and assessing algorithmic solutions to bias in ML. However, there is a disconnect between these solutions and practitioners' needs. By interviewing 30 ML practitioners of diverse backgrounds across 16 countries, and presenting them with a simulated use case, our study aims to investigate common fairness practices among professionals and how these are influenced by their backgrounds. The results reveal a superlative disparity among academia and industry practitioners. We also identify different practices in fairness and data exploration stages, influenced by the educational background as well as the level of experience of practitioners. Our study also finds how demographics have an impact on several aspects, such as willingness to accept and support legal actions taken against ML discrimination. In accordance with our findings, we suggest several actions that can be taken to improve fairness solutions, and we also highlight future directions for fairness research that can cause a positive impact on the way fairness is perceived by practitioners.
... Read more

The ability to identify and mitigate various risks and harms of using Machine Learning models in industry is an essential task. Specifically because these may produce harmful outcomes for stakeholders, including unfair or discriminatory results. Due to this there has been substantial research into the concepts of fairness and its metrics, bias and its mitigation, and algorithmic harms and their sources. Various toolkits have been created to guide practitioners to reflect on these topics and provide suggestions on algorithmic solutions to mitigate these risks. However, it is not yet known how widely these toolkits are used and how they are perceived in terms of usefulness. In this project, practitioners were interviewed in order to determine to what extend do envisioned practices of practitioners without experience with fairness toolkits differ from those with the experience. The two toolkits considered were the IBM AI Fairness360 and Microsoft FairLearn. The data collected from the interviews suggests that there could be fewer differences in practices of practitioners with experience and without experience with toolkits, than those with training or work roles in ethics and fairness in ML and those without. This suggests that experience the toolkit itself is not indicative of a more thorough approach to identifying and mitigating harms in fair Machine Learning. ... Read more

Despite the ever-growing advances in artificial intelligence (AI), common sense acquisition and reasoning is still comparingly in their early stages to other fields in AI. To further advance this field, it is necessary to collect large amounts of common sense facts or tacit knowledge to train such AI models. One effective way is to use crowdsourcing and games to make the collection process both widely available and fun at the same time. Currently, there are no tools to collect discriminative tacit knowledge efficiently and accurately. In this work, we propose FindItOut: a crowdsourcing tacit knowledge elicitation solution with multiplayer gamification and images to collect general-purpose and discriminative tacit knowledge. We show that using FindItOut, we can both efficiently and accurately collect discriminative tacit knowledge while also being fun and engaging. ... Read more

Tacit knowledge, unlike explicit knowledge, is not easily codifiable, yet important for machine learning models. This research explores a method to gather tacit knowledge about humor using a simple text-based party game, building on the existing idea of using games to gather tacit knowledge from crowds of people. Players propose prompts, which will then be answered by other players. They will then vote to determine which of the two answers to each prompt is the funniest. The engagement of the players with the game is measured and tacit knowledge is obtained from the jokes. With a large and diverse enough group of participants across games, a variety of tacit knowledge can be extracted. ... Read more

The manual process of collecting and labelling data required for machine learning tasks is labour-intensive, expensive, and time consuming. In the past, efforts have been made to crowdsource this data by either offering people monetary incentives, or by using a gamified approach where users contribute to databases as a side-effect of playing an enjoyable game. However, most of these efforts focus on using a competitive setting to incentivize players. This sometimes results in users spamming the dataset for personal gains. Research is lacking in how a collaborative setup, where players work together to make decisions by consensus, can be used to source knowledge that is more accurate and reliable. This paper describes the design and evaluation of SceneFinder, a game that aims to crowdsource reliable and diverse textual data about scenes (such as rooms, parks, monuments, etc) and the tacit knowledge relevant to them, such as information about their contents, their purpose and their surroundings. SceneFinder makes use of a collaborative setup that elicits a relevance based ranking of facts about these scenes, that distinguishes it from existing games in the field. ... Read more

Machine learning can still make harmful mistakes. A solution would be tacit knowledge. Machine learning needs this type of knowledge to improve. An example of such knowledge that can help make the system draw better logical conclusions would be: if presented with an open fridge, then it could deduct that the food will go bad. Tacit knowledge or common-sense knowledge refers to the type of knowledge which is acquired through experience, the kind only humans can create. GWAPs (game with a purpose) have shown quite promising results for acquiring such knowledge. Unfortunately, it could still contain errors due to users who only want to harm the game data, etc. and there is no method for validating such knowledge without involving humans somehow. Therefore, from the previously stated problem, our goal has emerged - develop a method for validating an existing data set and for later training machine learning models using a GWAP. There has been work done before using GWAPs to elicit such information, yet they are limited in the sense that their main focus is set on data collection, not validation. Since very few projects looked into it, we decided to investigate a new GWAP which has as main purpose tacit knowledge validation. The main question which we aim to answer is "How can we elicit and validate tacit knowledge using a game with the following settings: single-player, textual concepts, goal: associate words with their concepts." The game presents hints to the users and they have to guess, as fast as possible and with the least amount of tries as possible, which answer is correct from the 6 options that are provided. The evaluation of the game will be made using standard metrics such as games played, time spent playing, number of users, etc. The conclusion is that the GWAP, even with the lack of data, was quite capable of analyzing the quality of the data set and reached a conclusion that is easily confirmed by a mere look over the initial data set. ... Read more

This thesis looks at how to characterize weaknesses in machine learning models that are used for detecting privacy-sensitive data in images with the help of crowdsourcing. Before we can come up with a method to achieve a goal, we first need to make clear what we consider privacy-sensitive data. We took the General Data Protection Regulation (GDPR) as a starting point, and performed a crowdsourcing task to see how workers interpret this regulation. Interpreting legal texts can be difficult, there is room for interpretation and the perception of a legal text can change over time. Therefore, we need to take the input of the crowd, next to our own input, to operationalize this regulation to use in this context. Next, we took a machine learning model for detecting privacy-sensitive data in images in order to retrieve saliency maps, which helps us with explaining the inner-working of the model. Subsequently, the saliency maps are inspected through a crowdsourcing task, with the established privacy definition, to find out the strengths and weaknesses. From the results, we see that crowd workers can be efficiently used to find the strengths and weaknesses of a machine learning model, while keeping the privacy definition in mind. Workers are able to consistently apply their views about privacy across different images, whilst also increasing the trust people have in the machine learning model. This shows us that we can use crowdsourcing efficiently in a fairly difficult context of privacy, and paves the way for a more sophisticated approach to privacy-sensitive elements in images, and even for contexts other than privacy. ... Read more

To study how to involve the end-users in the development of machine learning explainability, this project has chosen the context of bird species identification. It intends to develop a platform where the end-users can learn bird knowledge while contributing to building the explainability of machine learning models. Among all the methods that equips machine learning models with explainability, this project adopts a framework called SECA (Semantic Concept Extraction and Analysis). In this framework, we require human-made-annotations to be made to the saliency maps of training photos to provide semantically understandable explanations to the end-users. On the other hand, we hope that the process of making annotations will also benefit the human annotators’ skills in bird species identification, in order to motivate their participation. Two main goals of the user research were: to understand the users’ needs for learning and to know their capability in making the annotations needed by the project owners. The user research started with qualitative and quantitative research to understand the current practices of the bird hobbyists, to define the target user groups, which were the birders with zero or little expertise. Then, in order to link their learning needs to the capability of machine learning explanations, three prototypes were built to collect their feedback. It was found out that they didn’t care much about the justification or transparency of bird ID apps, compared to learning knowledge in distinguishing birds. Then came the annotation test when we found the participants were able to finish the annotation task with high correctness (>93% on average). And the most popular annotations of each task were 100% correct. Finally, we built a functional, high-fidelity prototype with experiential interfaces and interactions, and tested it among 3 of the target users. They had positive feedback on the prototype’s usability and the overall workflow, which proved the feasibility of our concepts. Recommendations on usability were drawn at the end of this test. Throughout the research and design phases in this project, we have developed an approach to involve end-users in the annotation process of an explainable bird species identification model for their own benefit of fun and learning, which could potentially be applied to broader deployments ... Read more

Deep learning models have achieved state-of-the-art performance on several image classification tasks over the past years. Several studies claim to approach or even surpass human-levels of performance when using such models to classify images. However, these architectures are notoriously complex, thus making their interpretation a challenge. This limited interpretability, in turn, leads to several issues, such as restricting their applicability to critical domains like health care and finance.

Several methods in literature attempt to address this issue by providing local explanations which describe individual predictions or global ones that explain the model behaviour for a specific class. When focusing on global methods, we notice that they are limited with respect to the interpretability queries that they answer. For instance, consider we want to query whether the simultaneous presence of two objects is associated with predicting a specific class. To the best of our knowledge, there is no existing method that can tackle such a query type due to their limited expressivity. In this thesis, we address this limitation by answering the following research question: to what extent can image classification models be interpreted by analysing semantic features extracted from groups of salient image pixels?

We begin our study by investigating existing research work to devise the ideal characteristics that an interpretability method should adhere to. Our analysis highlights the aforementioned gap regarding the query complexity that existing methods cover. To address this limitation, we propose a new global interpretability method called SEmantic Feature Analysis (SEFA). To elaborate, it combines explanations of individual image predictions with semantic descriptions provided by human annotators about them, thus extracting the aforementioned semantic features. We argue that by analysing a structured data representation extracted out of semantic features will allow us to answer a wider range of interpretability queries compared to existing methods. The proposed method poses several challenges, such as identifying the number of image annotations required to obtain reliable results at a reasonable annotation cost.

Our results show that SEFA provides its users with the flexibility to answer several types of interpretability queries, including the ones that we found existing methods to be lacking. Further experimentation on its hyperparameters using three separate image classification tasks provides us with a set of suggested settings that one should use on similar datasets. Finally, we showcase the ability of SEFA to output semantic features relevant to the model classification behaviour by fine-tuning existing model architectures on biased datasets and evaluating whether the salient semantic features output describe the previous bias. ... Read more