InceptionTime neural network models were trained to detect distractions in manual control tasks with pursuit and preview displays. Training and test data were collected in an experiment where ten participants were deliberately distracted from the primary control task using the Surrogate Reference Task. Overall, distractions are easier to detect in pursuit tasks, with test accuracies of around 80% and 60% for pursuit and preview data, respectively. With preview, human controllers see the future target trajectory, which enables them to mitigate distraction effects. Unexpectedly, data with longer distractions from ‘hard’ secondary tasks are more difficult to classify than ‘easy’ distractions; an effect attributed to differences in human behavior between the training and test data collection conditions. These results show clear opportunities for neural network models to detect distractions, in real-time, for increasing safety of human-operated vehicles.