Reconstruction of Pilot Behaviour from Cockpit Image Recorder

Abstract

A method to automatically identify pilot actions from cockpit camera footage is reported in this paper. Although they have long been considered for the enhancement of flight safety, cockpit image recorders have not yet been standard equipment in aircraft cockpits. The rules on Flight Data Recorders have been changed, however, to include a cockpit image recorder as one of the safety devices, and it is recommended to be installed in small aircraft as a substitute for a Flight Data Recorder. With cockpit images becoming available, it would surely be useful for accident analysis as well as for daily flight analysis. Especially for the latter purpose, pilot behavior should be automatically analyzed and classified into specific actions, or procedures. The authors conducted a study to assess the feasibility of automatic detection of pilot actions in the cockpit by a machine learning process. Results show that even with a small amount of training data, the resulting algorithm could identify some typical actions, such as manipulation of the switches on the glare shield, with 80% accuracy. Even in cases with a button and a switch positioned very close to each other, the actions ‘pushing the switch’ and ‘pushing the button’ could be distinguished by the algorithm. The action estimation accuracy improves up to 90% when using the training data focused on the pilot’s body parts, rather than the data focused on the whole body.