How should drivers’ use of Automated Lane Keeping Systems (ALKS) be assessed? A study with experienced driving assessors in a Wizard-of-Oz vehicle

Abstract

This study aims to contribute to guidelines for driver licensing organizations on assessing driver competence in using Level 3 Automated Lane Keeping Systems (ALKS), based on an on-road experiment with eight professional driving assessors (i.e., expert driving examiners who train examiner candidates; 6 males, 2 females, all driving more than 20,000 km per year) in a Wizard-of-Oz vehicle. Using a think-aloud protocol, we captured cognitive processes during system supervision and take-over requests (TORs) in real-world traffic jams. A large language model (LLM)-based thematic analysis of transcripts revealed five themes: (1) Requirement for immediate environmental assessment, (2) Requirement for causal understanding, (3) Requirement for proactive intervention to maintain traffic flow, (4) Requirement for continuous “supervisor” engagement, and (5) Physical ergonomics and mode awareness. These findings indicate that, at least during short-duration usage, drivers do not simply rely on the system to disengage from driving; instead, they maintain active monitoring, physical readiness, and anticipatory skills. These observations blur the distinction between Level 2 and Level 3 automation, as the expert participants in this study generally remained attentive rather than adopting the ‘mind-off’ state that Level 3 theoretically allows. In conclusion, assessing ALKS usage involves not only evaluating a driver’s reaction to a TOR but also judging their performance as a systems manager responsible for anticipating conflicts and smoothly executing control transitions.