Participant feedback in previous research indicates a need for pilot training to handle non-routine situations with automation surprises. Therefore, we tested the effectiveness of using active problem solving during training on subsequent performance while dealing with automation surprises. We simulated a glass cockpit of a general aviation aircraft in a full motion flight simulator. An experimental group of private pilots (n = 10) was trained to actively diagnose and solve problems related to the autopilot, without foreknowledge of the training scenario. A control group (n = 10) received the same training scenarios with foreknowledge. The effectiveness of these pilots in dealing with both new and trained automation surprises was compared between groups. It was expected that the experimental group would be more effective in dealing with the new automation surprises, whereas the control group would be more effective in dealing with the repeated automation surprises. The experimental group indeed responded somewhat faster to the new automation surprises and were on average able to maintain the highest level of automation for a longer time than the control group. The opposite is true for the repeated automation surprises, where the control group was somewhat more effective in dealing with the automation surprises. Although these differences were of medium effect sizes (Cohens d = 0.5), they were not significant. Thus, the training approach taken in this study may need to be further enhanced and tested with more participants in order to have a significant effect.