How should drivers’ use of Automated Lane Keeping Systems (ALKS) be assessed? A study with experienced driving assessors in a Wizard of Oz vehicle

Bazilinskyy, P., Heikoop, D., Verstegen, R., Martens, M. H., De Winter, J. C. F.,

Submitted for publication

ABSTRACT This study aims to contribute to guidelines for driver licensing organizations on assessing driver competence in using Level 3 Automated Lane Keeping Systems (ALKS), based on an on-road experiment with eight professional driving assessors (i.e., expert driving examiners who train experienced drivers to become driving examiners of their own) in a Wizard of Oz vehicle. Using a think-aloud protocol, we captured cognitive processes during system supervision and take-over requests (TORs) in real-world traffic jams. An LLM-based thematic analysis of transcripts revealed four themes: (1) Continuous situational awareness as a prerequisite for trust and action, (2) The need for system transparency and predictability, (3) Proactive manual control to ensure safety and traffic flow, and (4) The take-over action as an immediate, reflexive response. These results suggest that safe ALKS operation requires active monitoring and anticipatory skills. This blurs the distinction between Level 2 and Level 3 automation, as the expert participants in our study generally remained attentive rather than adopting the ‘mind-off’ state that Level 3 theoretically allows. In conclusion, assessing ALKS usage involves not only evaluating a driver’s reaction to a TOR but also judging their performance as a systems manager responsible for anticipating conflicts and smoothly executing control transitions.