Investigating fair rankers under the expected exposure framework

Investigating fair rankers under the expected exposure framework

Visser, Maaike (TU Delft Electrical Engineering, Mathematics and Computer Science)

Hauff, C. (mentor)
Chen, Lydia Y. (graduation committee)

Delft University of Technology

Computer Science | Data Science and Technology

2022-09-26

As the amount of information available in the world grows, Information Retrieval (IR) systems have become an integral part of day to day life. They determine what subset of the large pool of information is shown to people. IR algorithms determine which items should be returned in response to a query and rank the results in a ranked list.
Recently, concerns about the fairness of IR algorithms have surfaced. In particular, research is being done into whether IR algorithms are fair to producers, people or organizations that provide the items that are retrieved by IR algorithms. The higher an item is in the ranked list, the more attention it receives from users. This attention translates to benefits for the producers, e.g. fame or financial compensation.
In this thesis we investigate the fairness of IR algorithms in terms of a specific measure for provider fairness: the Expected Exposure Loss (EEL). This measure measures whether the providers of equally relevant items receive the same amount of attention in expectation. EEL was first proposed as part of the 2020 TREC Fair Ranking track (FAIR-TREC), which also provided a matching dataset. We investigate for two IR systems whether they achieve fairness on this dataset. We conduct a failure analysis and propose improvements for both systems.
We find that for a system that always returns the same ranking it is not useful to improve its accuracy, but rather that it benefits most from fairness-aware post-processing. By contrast, a fairness-aware systems does benefit from a higher accuracy, since EEL requires that equally relevant items are treated the same. We note that the generalizability of our investigation is limited due to the small size of the FAIR-TREC 2020 dataset and recommend that a larger dataset be made available.

information retrieval
fairness
expected exposure

http://resolver.tudelft.nl/uuid:329d6116-35ef-453d-a2e8-874ce6a99337

https://github.com/pilmus/thesis Repository for reproduction purposes.

Student theses

master thesis

© 2022 Maaike Visser