This Is Not What We Ordered: Exploring Why Biased Search Result Rankings Affect User Attitudes on Debated Topics

This Is Not What We Ordered: Exploring Why Biased Search Result Rankings Affect User Attitudes on Debated Topics

Draws, T.A. (TU Delft Web Information Systems)
Tintarev, N. (Universiteit Maastricht)
Gadiraju, Ujwal (TU Delft Web Information Systems)
Bozzon, A. (TU Delft Human-Centred Artificial Intelligence)
Timmermans, B. (IBM)

2021

In web search on debated topics, algorithmic and cognitive biases strongly influence how users consume and process information. Recent research has shown that this can lead to a search engine manipulation effect (SEME): when search result rankings are biased towards a particular viewpoint, users tend to adopt this favored viewpoint. To better understand the mechanisms underlying SEME, we present a pre-registered, 5 x 3 factorial user study investigating whether order effects (i.e., users adopting the viewpoint pertaining to higher-ranked documents) can cause SEME. For five different debated topics, we evaluated attitude change after exposing participants with mild pre-existing attitudes to search results that were overall viewpoint-balanced but reflected one of three levels of algorithmic ranking bias. We found that attitude change did not differ across levels of ranking bias and did not vary based on individual user differences. Our results thus suggest that order effects may not be an underlying mechanism of SEME. Exploratory analyses lend support to the presence of exposure effects (i.e., users adopting the majority viewpoint among the results they examine) as a contributing factor to users' attitude change. We discuss how our findings can inform the design of user bias mitigation strategies.

ranking bias
user attitudes
user-centered evaluation
web search

http://resolver.tudelft.nl/uuid:7a7259ff-2e93-4970-8e66-23a134d77092

https://doi.org/10.1145/3404835.3462851

Association for Computing Machinery (ACM), New York, NY, USA

9781450380379

SIGIR 2021 - Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021-07-11 → 2021-07-15, Online, Monteal, Canada

SIGIR 2021 - Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

Institutional Repository

conference paper

© 2021 T.A. Draws, N. Tintarev, Ujwal Gadiraju, A. Bozzon, B. Timmermans