Every human makes mistakes: Exploring the sensitivity of deep-learned object detectors to human annotation noise

Abstract

The annotation effort associated with object detection is extremely costly. One option to reduce cost is to relax the demands on annotation quality, effectively allowing annotation noise. Current research primarily focuses on noise correction before or during training. However, there remains a gap in the research regarding the impact of specific types of human annotation noise on object-detector performance. This research aimed to determine how sensitive object detectors are to human annotation noise. A systematic methodology was developed to generate and quantify the effects of four noise types: missing annotations, extra annotations, inaccurate bounding boxes, and wrong classification labels. Additionally, evaluations were conducted on YOLOv8 and Faster R-CNN using the PASCAL VOC 2012, VisDrone, and Brain-Tumor datasets. The experiments demonstrated that adding noise to smaller datasets adversely affects the performance of object detectors trained on these datasets more than it does for those trained on larger datasets. Similarly, annotation noise in small objects affects detector performance more than large objects. Furthermore, YOLOv8 is resilient to low levels of missing annotations and inaccurate bounding boxes but is sensitive to all levels of incorrect classification labels. Interestingly, extra annotations seem to have a regularization effect on YOLOv8. In contrast, Faster R-CNN is generally more susceptible to annotation noise compared to YOLOv8, particularly concerning extra annotations, though both models display similar trends regarding inaccurate bounding boxes.