Print Email Facebook Twitter Investigating Data Collection and Reporting Practices of Human Annotations in Societally Impactful Machine Learning Applications Title Investigating Data Collection and Reporting Practices of Human Annotations in Societally Impactful Machine Learning Applications: A Systematic Review of Top-Cited IEEE Access Papers Author Ibrahim, Ahmed (TU Delft Electrical Engineering, Mathematics and Computer Science) Contributor Liem, C.C.S. (mentor) Demetriou, A.M. (mentor) Broz, F. (graduation committee) Degree granting institution Delft University of Technology Programme Computer Science and Engineering Project CSE3000 Research Project Date 2023-06-28 Abstract This systematic review investigates the practices and implications of human annotations in machine learning (ML) research. Analyzing a selection of 100 papers from the IEEE Access Journal, the study explores the data collection and reporting methods employed. The findings reveal a prevalent lack of standardization and formalization in the annotation process. Key details such as annotation sources, number of annotators, and formal instructions are frequently neglected, possibly compromising the quality and effectiveness of ML algorithms. Domain-specific implications are discussed, highlighting the need for comprehensive annotation practices in areas like medical diagnostics, language processing, and intelligent vehicle systems. The study contributes to the field by emphasizing the importance of standardized procedures and transparency in ML research. Future research is recommended to develop systematic annotation methodologies and examine the impact of subpar annotation on data quality. Subject Huamn AnnotationMachine LearningIEEE Access To reference this document use: http://resolver.tudelft.nl/uuid:645945a4-0f7b-41f5-ae66-de2417783add Part of collection Student theses Document type bachelor thesis Rights © 2023 Ahmed Ibrahim Files PDF CSE3000_Research_Project_ ... ion_2_.pdf 212.23 KB Close viewer /islandora/object/uuid:645945a4-0f7b-41f5-ae66-de2417783add/datastream/OBJ/view