Validity and reliability of naturalistic driving scene categorization Judgments from crowdsourcing

Validity and reliability of naturalistic driving scene categorization Judgments from crowdsourcing

Cabrall, C.D.D. (TU Delft Intelligent Vehicles)
Lu, Z. (TU Delft Intelligent Vehicles)
Kyriakidis, M. (TU Delft Biomechatronics & Human-Machine Control; ETH Zürich)
Manca, L. (TU Delft Biomechatronics & Human-Machine Control)
Dijksterhuis, C. (TU Delft Biomechatronics & Human-Machine Control; Hanze Hogeschool Groningen)
Happee, R. (TU Delft Intelligent Vehicles)
de Winter, J.C.F. (TU Delft Biomechatronics & Human-Machine Control)

2018

A common challenge with processing naturalistic driving data is that humans may need to categorize great volumes of recorded visual information. By means of the online platform CrowdFlower, we investigated the potential of crowdsourcing to categorize driving scene features (i.e., presence of other road users, straight road segments, etc.) at greater scale than a single person or a small team of researchers would be capable of. In total, 200 workers from 46 different countries participated in 1.5. days. Validity and reliability were examined, both with and without embedding researcher generated control questions via the CrowdFlower mechanism known as Gold Test Questions (GTQs).By employing GTQs, we found significantly more valid (accurate) and reliable (consistent) identification of driving scene items from external workers. Specifically, at a small scale CrowdFlower Job of 48 three-second video segments, an accuracy (i.e., relative to the ratings of a confederate researcher) of 91% on items was found with GTQs compared to 78% without. A difference in bias was found, where without GTQs, external workers returned more false positives than with GTQs. At a larger scale CrowdFlower Job making exclusive use of GTQs, 12,862 three-second video segments were released for annotation. Infeasible (and self-defeating) to check the accuracy of each at this scale, a random subset of 1012 categorizations was validated and returned similar levels of accuracy (95%).In the small scale Job, where full video segments were repeated in triplicate, the percentage of unanimous agreement on the items was found significantly more consistent when using GTQs (90%) than without them (65%). Additionally, in the larger scale Job (where a single second of a video segment was overlapped by ratings of three sequentially neighboring segments), a mean unanimity of 94% was obtained with validated-as-correct ratings and 91% with non-validated ratings. Because the video segments overlapped in full for the small scale Job, and in part for the larger scale Job, it should be noted that such reliability reported here may not be directly comparable. Nonetheless, such results are both indicative of high levels of obtained rating reliability.Overall, our results provide compelling evidence for CrowdFlower, via use of GTQs, being able to yield more accurate and consistent crowdsourced categorizations of naturalistic driving scene contents than when used without such a control mechanism. Such annotations in such short periods of time present a potentially powerful resource in driving research and driving automation development.

Crowdsourcing
Dash cam
Data annotation
Naturalistic driving study
Reliability
Validity

http://resolver.tudelft.nl/uuid:8108eb07-ea80-42f7-9bad-5c027e629208

https://doi.org/10.1016/j.aap.2017.08.036

2018-03-12

0001-4575

Accident Analysis & Prevention, 114, 25-33

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Institutional Repository

journal article

© 2018 C.D.D. Cabrall, Z. Lu, M. Kyriakidis, L. Manca, C. Dijksterhuis, R. Happee, J.C.F. de Winter