Print Email Facebook Twitter What Should You Know? A Human-In-the-Loop Approach to Unknown Unknowns Characterization in Image Recognition Title What Should You Know? A Human-In-the-Loop Approach to Unknown Unknowns Characterization in Image Recognition Author Sharifi Noorian, S. (TU Delft Web Information Systems) Qiu, S. (TU Delft Web Information Systems) Gadiraju, Ujwal (TU Delft Web Information Systems) Yang, J. (TU Delft Web Information Systems) Bozzon, A. (TU Delft Sustainable Design Engineering; TU Delft Human-Centred Artificial Intelligence) Department Sustainable Design Engineering Date 2022 Abstract Unknown unknowns represent a major challenge in reliable image recognition. Existing methods mainly focus on unknown unknowns identification, leveraging human intelligence to gather images that are potentially difficult for the machine. To drive a deeper understanding of unknown unknowns and more effective identification and treatment, this paper focuses on unknown unknowns characterization. We introduce a human-in-the-loop, semantic analysis framework for characterizing unknown unknowns at scale. We engage humans in two tasks that specify what a machine should know and describe what it really knows, respectively, both at the conceptual level, supported by information extraction and machine learning interpretability methods. Data partitioning and sampling techniques are employed to scale out human contributions in handling large data. Through extensive experimentation on scene recognition tasks, we show that our approach provides a rich, descriptive characterization of unknown unknowns and allows for more effective and cost-efficient detection than the state of the art. Subject humans in the loopsemantic analysisUnknown unknowns To reference this document use: http://resolver.tudelft.nl/uuid:2e9e89cf-f216-4a2e-94ea-19ec053ec342 DOI https://doi.org/10.1145/3485447.3512040 Publisher Association for Computing Machinery (ACM) ISBN 978-1-4503-9096-5 Source WWW 2022 - Proceedings of the ACM Web Conference 2022 Event 31st ACM World Wide Web Conference, WWW 2022, 2022-04-25 → 2022-04-29, Virtual, Online at Lyon, France Series WWW 2022 - Proceedings of the ACM Web Conference 2022 Part of collection Institutional Repository Document type conference paper Rights © 2022 S. Sharifi Noorian, S. Qiu, Ujwal Gadiraju, J. Yang, A. Bozzon Files PDF 3485447.3512040.pdf 9.7 MB Close viewer /islandora/object/uuid:2e9e89cf-f216-4a2e-94ea-19ec053ec342/datastream/OBJ/view