Systematic experimental investigations were conducted to study the microstructures and impact toughness of each heat affected zone (HAZ) formed during rail flash-butt welding. A high-strength carbide-free bainitic rail steel was subjected to different thermal simulation cycles to separately reproduce each HAZ subzone by tailoring the peak temperature (PT) with respect to 700, 850, 920, 1000 and 1350 °C, and hence to generate the corresponding microstructures by using Gleeble-3500 simulator. Results show that the HAZ subzones exhibit complicated microstructures depending on the PTs, and with increasing PT the dominant bainitic microstructure type evolves from polygonal bainitic ferrite (700 °C) to a mixture of fine bainitic ferrite and granular bainite (850–1000 °C), and finally to coarse bainitic ferrite and granular bainite (1350 °C). Impact tests demonstrate that the impact toughness initially increases significantly as the PT reaches 920 °C (i.e., fine-grained HAZ), beyond which the impact toughness starts to decrease. The fine-grained HAZ displays optimal impact toughness in HAZs, yet which is lower than the base metal. Moreover, the morphology and distribution of martensite-austenite (M-A) constituents is strongly dependent on the welding PT, and the high fraction blocky and coarse slender M-A constituents is considered to be detrimental for the impact toughness.