Print Email Facebook Twitter Perspective Title Perspective: Leveraging Human Understanding for Identifying and Characterizing Image Atypicality Author Sharifi Noorian, S. (TU Delft Web Information Systems) Qiu, S. (Hunan Institute of Advanced Technology) Sayin, Burcu (Università di Trento) Balayn, A.M.A. (TU Delft Web Information Systems) Gadiraju, Ujwal (TU Delft Web Information Systems) Yang, J. (TU Delft Web Information Systems) Bozzon, A. (TU Delft Human-Centred Artificial Intelligence) Date 2023 Abstract High-quality data plays a vital role in developing reliable image classification models. Despite that, what makes an image difficult to classify remains an unstudied topic. This paper provides a first-of-its-kind, model-agnostic characterization of image atypicality based on human understanding. We consider the setting of image classification "in the wild", where a large number of unlabeled images are accessible, and introduce a scalable and effective human computation approach for proactive identification and characterization of atypical images. Our approach consists of i) an image atypicality identification and characterization task that presents to the human worker both a local view of visually similar images and a global view of images from the class of interest and ii) an automatic image sampling method that selects a diverse set of atypical images based on both visual and semantic features. We demonstrate the effectiveness and cost-efficiency of our approach through controlled crowdsourcing experiments and provide a characterization of image atypicality based on human annotations of 10K images. We showcase the utility of the identified atypical images by testing state-of-the-art image classification services against such images and provide an in-depth comparative analysis of the alignment between human- and machine-perceived image atypicality. Our findings have important implications for developing and deploying reliable image classification systems. Subject humans in the loopimage atypicalityImage classificationmachine learning in the wild To reference this document use: http://resolver.tudelft.nl/uuid:3266269f-cb93-4c28-8226-6580fff0a2f6 DOI https://doi.org/10.1145/3581641.3584096 Publisher Association for Computing Machinery (ACM), New York ISBN 979-8-4007-0106-1 Source IUI 2023 - Proceedings of the 28th International Conference on Intelligent User Interfaces Event 28th International Conference on Intelligent User Interfaces, IUI 2023, 2023-03-27 → 2023-03-31, Sydney, Australia Part of collection Institutional Repository Document type conference paper Rights © 2023 S. Sharifi Noorian, S. Qiu, Burcu Sayin, A.M.A. Balayn, Ujwal Gadiraju, J. Yang, A. Bozzon Files PDF 3581641.3584096.pdf 11 MB Close viewer /islandora/object/uuid:3266269f-cb93-4c28-8226-6580fff0a2f6/datastream/OBJ/view