Participant feedback in previous research indicates a need for pilot training to handle non-routine situations with automation surprises. Therefore, we tested the effectiveness of using active problem solving during training on subsequent performance while dealing with automation surprises. We simulated a glass cockpit of a general aviation aircraft in a full motion flight simulator. An experimental group of private pilots (n = 10) was trained to actively diagnose and solve problems related to the autopilot, without foreknowledge of the training scenario. A control group (n = 10) received the same training scenarios with foreknowledge. The effectiveness of these pilots in dealing with both new and trained automation surprises was compared between groups. It was expected that the experimental group would be more effective in dealing with the new automation surprises, whereas the control group would be more effective in dealing with the repeated automation surprises. The experimental group indeed responded somewhat faster to the new automation surprises and were on average able to maintain the highest level of automation for a longer time than the control group. The opposite is true for the repeated automation surprises, where the control group was somewhat more effective in dealing with the automation surprises. Although these differences were of medium effect sizes (Cohens d = 0.5), they were not significant. Thus, the training approach taken in this study may need to be further enhanced and tested with more participants in order to have a significant effect.

Recent studies and accident investigations show the detrimental effects of startle and (automation) surprise on flight crew performance in terms of cognitive reasoning and sensemaking. Previous research conducted at Delft University of Technology shows a positive effect of training variability on crew performance in surprising tasks related to training, but in case of novel, unrelated surprising tasks, training methods remain to be investigated. Other recent studies and proposals suggest to use checklist-based training methods to prepare pilots for puzzling and surprising scenarios, but effects on flight crew performance have yet to be published. Therefore a simulator-based, between-subjects experiment was conducted to explore the effects of a four-item checklist-based training on pilot performance. Training elements focused on managing stress and enhancing situation awareness, as well as improving sensemaking and decision-making processes. It was expected that trained pilots would exhibit better overall performance in test scenarios involving off-nominal, surprising situations. Nevertheless, results indicated no significant difference in pilot response to an initial disturbance, but did show significant performance improvement in scenarios involving a second, ensuing upset compared to the baseline group. This suggests that checklist-based training methods offer a structured problem-solving approach, resulting in better response to consecutive disturbances in select scenarios.