Ensuring the safety and longevity of railway bridges requires efficient, non-invasive methods for monitoring their health and detecting structural damage. Drive-by health monitoring (DBHM) has emerged as a promising approach, using vehicle-mounted sensors, such as axle box acceleration (ABA), to assess the structural integrity of bridges. This method offers the advantage of frequent monitoring under operational conditions. However, DBHM faces challenges in real-world applications due to the subtle influence of local damage and disturbances like vehicle dynamics, track irregularities, and noise. This study investigates the feasibility of using ABA to detect structural damage in a real railway bridge. Continuous wavelet transforms and filtering techniques are used to isolate different vibration components within ABA signals. A finite element model of a cracked beam is developed, and simulations reveal that local structural damage introduces a small, local peak in the quasi-static ABA component. Field measurements show the variability of ABA measurements over space and time and the resulting difficulty in directly detecting the local damage. However, probabilistic analysis suggests that reference signals under healthy conditions, combined with frequent monitoring, can enhance the reliability of damage detection using DBHM.