A Human-In-the-Loop Framework to Assess Multimodal Machine Learning Models

A Human-In-the-Loop Framework to Assess Multimodal Machine Learning Models

Chen, Dina (TU Delft Electrical Engineering, Mathematics and Computer Science)

Yang, J. (mentor)
Houben, G.J.P.M. (graduation committee)
Lan, G. (graduation committee)
Tocchetti, A. (mentor)

Delft University of Technology

Computer Science

2022-11-30

Recent works explain the DNN models that perform image classification tasks following the "attribution, human-in-the-loop, extraction" workflow. However, little work has looked into such an approach for explaining DNN models for language or multimodal tasks. To address this gap, we propose a framework that explains and assesses the model utilizing both the categorical/numerical features and the text while optimizing the "attribution, human-in-the-loop, extraction" workflow. In particular, our framework deals with limited human resources, especially when domain experts are required for human-in-the-loop tasks. It provides insight regarding which set of data should the human-in-the-loop tasks be brought in. We share the results of applying this framework to a multimodal transformer that performs text classification tasks for compliance detection in the financial context.

XAI
Multimodal
NLP
Human-in-the-loop

http://resolver.tudelft.nl/uuid:806e001d-9bf3-49e1-b2f4-298c747aea2a

Student theses

master thesis

© 2022 Dina Chen