To fully optimize the synergy between human operators and machines in modern day’s highly automated vehicle control tasks, a real-time quantitative feedback of skill level is required. Direct feedback of skill level could be used to enable scalable levels of autonomy of the controlled system, or to provide a warning when sudden skill level deterioration is detected, improving safety. Cybernetics has proven to be a useful tool to assess pilot skill level, but most traditional methods suffer from the fundamental issue of assuming the human controller to be constant over time, ignoring subtle changes in control behavior. Employing deep artificial neural networks directly to raw time series of control behavior may be a solution to this problem. Using a deep residual convolutional neural network (ResNet) architecture, this research shows that 1.2 second windows of experimental human control data—from a previously conducted compensatory tracking experiment in the SIMONA Research Simulator at Delft University of Technology—can be classified as either ’skilled’ or ’unskilled’ with an average validation accuracy of 92% in a moving-base setting and 88% in a fixed-base setting. Results indicate that the trained network is not a one-size-fits-all classifier in its current state, as the skill levels of isolated subjects with off-nominal learning curves are difficult to predict. Introduction of SHapley Additive exPlanations and class visualizations with activation maximization add transparency to the trained classifier’s predictions, offering a new perspective on distinctive characteristics of manual control skill level in the time domain. This explainable deep learning approach to skill level identification enables real-time quantitative evaluation of control behavior, opening a new realm of possibilities to enhance safety in automated systems that rely on smooth interaction with the human operator.