Title
TaskGenie: Crowd-Powered Task Generation for Struggling Search
Author
Xu, Luyan (Renmin University of China)
Zhou, Xuan (East China Normal University)
Gadiraju, Ujwal (TU Delft Web Information Systems) 
Contributor
Huang, Zhisheng (editor)
Beek, Wouter (editor)
Wang, Hua (editor)
Zhang, Yanchun (editor)
Zhou, Rui (editor)
Date
2020
Abstract
Search tasks provide a medium for the evaluation of system performance and the underlying analytical aspects of IR systems. Researchers have recently developed new interfaces or mechanisms to support vague information needs and struggling search. However, little attention has been paid to the generation of a unified task set for evaluation and comparison of search engine improvements for struggling search. Generation of such tasks is inherently difficult, as each task is supposed to trigger struggling and exploring user behavior rather than simple search behavior. Moreover, the everchanging landscape of information needs would render old task sets less ideal if not unusable for system evaluation. In this paper, we propose a task generation method and develop a crowd-powered platform called TaskGenie to generate struggling search tasks online. Our experiments and analysis show that the generated tasks are qualified to emulate struggling search behaviors consisting of ‘repeated similar queries’ and ‘quick-back clicks’, etc. – tasks of diverse topics, high quality and difficulty can be created using this framework. For the benefit of the community, we publicly released the platform, a task set containing 80 topically diverse struggling search tasks generated and examined in this work, and the corresponding anonymized user behavior logs.
Subject
SERPs
User behavior
User interaction
Web search
To reference this document use:
http://resolver.tudelft.nl/uuid:90bd425f-220a-42d2-ba06-4c6091b77db1
DOI
https://doi.org/10.1007/978-3-030-62008-0_1
Publisher
Springer, Cham
Embargo date
2021-11-08
ISBN
978-3-030-62007-3
Source
Web Information Systems Engineering: WISE 2020 - Proceedings
Event
21st International Conference on Web Information Systems Engineering, WISE 2020, 2020-10-20 → 2020-10-24, Amsterdam, Netherlands
Series
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 0302-9743, 12343
Bibliographical note
Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.
Part of collection
Institutional Repository
Document type
conference paper
Rights
© 2020 Luyan Xu, Xuan Zhou, Ujwal Gadiraju