document

A Human-In-the-Loop Framework to Assess Multimodal Machine Learning Models

Chen, Dina (author)

Recent works explain the DNN models that perform image classification tasks following the "attribution, human-in-the-loop, extraction" workflow. However, little work has looked into such an approach for explaining DNN models for language or multimodal tasks. To address this gap, we propose a framework that explains and assesses the model...

master thesis 2022

document

CHIME: Causal Human-In-the-Loop Model Explanations

Biswas, Shreyan (author)

Explaining the behaviour of Artificial Intelligence models has become a necessity. Their opaqueness and fragility are not tolerable in high-stakes domains especially. <br/>Although considerable progress is being made in the field of Explainable Artificial Intelligence, scholars have demonstrated limits and flaws of existing approaches:...

master thesis 2022

document

Debugging Machine Learning on Time-series Data

Zheng, Meng (author)

Machine learning models are so-called a "black box," which means people can not easily observe the relationship between the output and input or explain the reason for such results. In recent years, much work has been done on interpretable machine-learning, such as Shapley values, counterfactual explanations, partial dependence plots, or saliency...

master thesis 2022

document

Characterizing the knowns and unknowns of text simplification models

WANG, Siwei (author)

master thesis 2022

document

A Human-In-the-Loop System for Interpreting Image Recognition Models

Ziengs, Bart (author)

Interpretability of ML models and image recognition models specifaclly, is a increasing problem. In this thesis, the design and implementation of Brickroutine: a system that used a trained model, is presented. Using human annotations, semantic interpretations are given to image classification problems. By giving an iterative approach in terms of...

master thesis 2022

document

Visually-guided motion planning for autonomous driving from interactive demonstrations

Perez Dattari, R.J. (author), Ferreira de Brito, B.F. (author), de Groot, O.M. (author), Kober, J. (author), Alonso Mora, J. (author)

The successful integration of autonomous robots in real-world environments strongly depends on their ability to reason from context and take socially acceptable actions. Current autonomous navigation systems mainly rely on geometric information and hard-coded rules to induce safe and socially compliant behaviors. Yet, in unstructured urban...

journal article 2022

document

What Should You Know? A Human-In-the-Loop Approach to Unknown Unknowns Characterization in Image Recognition

Sharifi Noorian, S. (author), Qiu, S. (author), Gadiraju, Ujwal (author), Yang, J. (author), Bozzon, A. (author)

Unknown unknowns represent a major challenge in reliable image recognition. Existing methods mainly focus on unknown unknowns identification, leveraging human intelligence to gather images that are potentially difficult for the machine. To drive a deeper understanding of unknown unknowns and more effective identification and treatment, this...

conference paper 2022

document

SparCAssist: A Model Risk Assessment Assistant Based on Sparse Generated Counterfactuals

Zhang, Zijian (author), Setty, Vinay (author), Anand, A. (author)

We introduce SparCAssist, a general-purpose risk assessment tool for the machine learning models trained for language tasks. It evaluates models' risk by inspecting their behavior on counterfactuals, namely out-of-distribution instances generated based on the given data instance. The counterfactuals are generated by replacing tokens in...

conference paper 2022

document

Using human-in-the-loop and explainable AI to envisage new future work practices

Tsiakas, K. (author), Murray-Rust, D.S. (author)

In this paper, we discuss the trends and challenges of the integration of Artificial Intelligence (AI) methods in the workplace. An important aspect towards creating positive AI futures in the workplace is the design of fair, reliable and trustworthy AI systems which aim to augment human performance and perception, instead of replacing them...

conference paper 2022

document

Enabling Human-In-The-Loop Interpretability Methods of Machine Learning Models: The Case of Bird Species Identification

Zeng, Wei (author)

To study how to involve the end-users in the development of machine learning explainability, this project has chosen the context of bird species identification. It intends to develop a platform where the end-users can learn bird knowledge while contributing to building the explainability of machine learning models. Among all the methods that...

master thesis 2021

document

MARTA: Leveraging Human Rationales for Explainable Text Classification

Arous, Ines (author), Dolamic, Ljiljana (author), Yang, J. (author), Bhardwaj, Akansha (author), Cuccu, Giuseppe (author), Cudré-Mauroux, Philippe (author)

Explainability is a key requirement for text classification in many application domains ranging from sentiment analysis to medical diagnosis or legal reviews. Existing methods often rely on "attention" mechanisms for explaining classification results by estimating the relative importance of input units. However, recent studies have shown that...

conference paper 2021

document

Towards a human in the loop approach to preserve privacy in images

Mauri, A. (author), Bozzon, A. (author)

Current artificial intelligence and information retrieval systems need to be trained with a large amount of data to achieve satisfying performance. A popular solution to create such datasets is to employ crowdsourcing; however, the content to be annotated may contain private or sensitive information that can be extracted by workers, limiting...

journal article 2021

document

A review and experimental analysis of active learning over crowdsourced data

Sayin, Burcu (author), Krivosheev, Evgeny (author), Yang, J. (author), Passerini, Andrea (author), Casati, Fabio (author)

Training data creation is increasingly a key bottleneck for developing machine learning, especially for deep learning systems. Active learning provides a cost-effective means for creating training data by selecting the most informative instances for labeling. Labels in real applications are often collected from crowdsourcing, which engages...

journal article 2021

document

Design, fabrication and validation of an ankle-foot orthosis simulator system to optimize mechanical characteristics of AFOs for humans with impaired locomotion

Wouterse, Lars (author)

The muscles around the ankle (calf and dorsal flexors) are essential for performing activities in daily life, like walking. Neurological and muscular pathologies, such as stroke, cerebral palsy, spinal cord injury, muscle atrophy and post-polio syndrome, affect the ability of voluntary muscle control and/or muscle strength of such muscles. This...

master thesis 2020

document

Non steady-state prediction of the instantaneous metabolic energy expenditure in pathological gait: an exploratory study

Filius, Suzanne (author)

BACKGROUND - In a variety of patients with locomotive disorders gait efficiency is often assessed using mobile metabolic gas analysis during rehabilitative interventions. Conventionally, these measurements take 6 minutes per evaluated condition (e.g. with/without ankle-foot-orthosis; AFO). This duration is required since there is a dynamical...

master thesis 2019

document

Validation methodology for expert-annotated datasets: Event annotation case study

Inel, O. (author), Aroyo, Lora (author)

Event detection is still a difficult task due to the complexity and the ambiguity of such entities. On the one hand, we observe a low inter-annotator agreement among experts when annotating events, disregarding the multitude of existing annotation guidelines and their numerous revisions. On the other hand, event extraction systems have a...

conference paper 2019

document

Coordination in Multi-Actor Self-Separation in Complex Airspace Environments: A human-in-the-loop experiment using the Solution Space Diagram

Tegginamani Shiva Kumar, Siddarth (author)

The Air Traffic Management research community expects that the responsibility of maintaining a minimum safe separation distance between aircraft will move from the groundbased Air Traffic Control, a centralized system, towards the flight deck, a decentralized system. Previous research on the decentralized, also known as distributed Air Traffic...

master thesis 2018

document

CaptureBias: Supporting Media Scholars with Ambiguity-Aware Bias Representation for News Videos

de Jong, M. (author), Mavridis, P. (author), Aroyo, Lora (author), Bozzon, A. (author), Vos, Jesse de (author), Oomen, Johan (author), Dimitrova, Antoaneta (author), Badenoch, Alec (author)

In this project we explore the presence of ambiguity in textual and visual media and its influence on accurately understanding and<br/>capturing bias in news. We study this topic in the context of supporting<br/>media scholars and social scientists in their media analysis. Our focus<br/>lies on racial and gender bias as well as framing and the...

conference paper 2018

document

Sensorimotor manipulations of the balance control loop-beyond imposed external perturbations

Rasman, Brandon G. (author), Forbes, P.A. (author), Tisserand, Romain (author), Blouin, Jean Sébastien (author)

Standing balance relies on the integration of multiple sensory inputs to generate the motor commands required to stand. Mechanical and sensory perturbations elicit compensatory postural responses that are interpreted as a window into the sensorimotor processing involved in balance control. Popular methods involve imposed external...

review 2018

document

A Human in the Loop Approach to Capture Bias and Support Media Scientists in News Video Analysis

Mavridis, P. (author), de Jong, M. (author), Aroyo, Lora (author), Bozzon, A. (author), Vos, Jesse de (author), Oomen, Johan (author), Dimitrova, Antoaneta (author), Badenoch, Alec (author)

Bias is inevitable and inherent in any form of communication. News often appear biased to citizens with dierent political orientations, and understood dierently by news media scholars and the broader public. In this paper we advocate the need for accurate methods for bias identication in video news item, to enable rich analytics capabilities in...

conference paper 2018