In recent years, CNN-based object detectors have been widely adopted in autonomous systems. Although their capabilities are employed across various industries, these detectors are inherently susceptible to adversarial attacks. Despite extensive studies on their effects on image classification, adversarial attacks remain largely unexplored in object detection. In particular, we note the reduced number of studies employing benchmarks for these types of attacks. Object detectors can be easily deceived by adding carefully devised perturbations to their inputs, rendering them unreliable. This study investigates the transferability of one such adversarial attack type, the Targeted Objectness Gradient (TOG), on different variations of the YOLO architecture to formally assess its vulnerability under different scenarios in the maritime domain. To investigate the significance of TOG adversarial attacks across variations of YOLO architectures and combinations of maritime datasets (all publicly available), we conducted a statistical analysis of black-box and white-box attacks. Our research questions were formulated to address a range of concerns that encompass various complexities to be considered in the detection of maritime objects. Our presented results underline the transferable nature of TOG adversarial attacks and the compelling need to benchmark such attacks in the maritime object detection domain.