Recent studies and accident investigations show the detrimental effects of startle and (automation) surprise on flight crew performance in terms of cognitive reasoning and sensemaking. Previous research conducted at Delft University of Technology shows a positive effect of training variability on crew performance in surprising tasks related to training, but in case of novel, unrelated surprising tasks, training methods remain to be investigated. Other recent studies and proposals suggest to use checklist-based training methods to prepare pilots for puzzling and surprising scenarios, but effects on flight crew performance have yet to be published. Therefore a simulator-based, between-subjects experiment was conducted to explore the effects of a four-item checklist-based training on pilot performance. Training elements focused on managing stress and enhancing situation awareness, as well as improving sensemaking and decision-making processes. It was expected that trained pilots would exhibit better overall performance in test scenarios involving off-nominal, surprising situations. Nevertheless, results indicated no significant difference in pilot response to an initial disturbance, but did show significant performance improvement in scenarios involving a second, ensuing upset compared to the baseline group. This suggests that checklist-based training methods offer a structured problem-solving approach, resulting in better response to consecutive disturbances in select scenarios.