A New Perspective on Score Standardization

A New Perspective on Score Standardization

Urbano, Julián (TU Delft Multimedia Computing)
De Lima, H.A. (TU Delft Multimedia Computing)
Hanjalic, A. (TU Delft Intelligent Systems)

Intelligent Systems

2019

In test collection based evaluation of IR systems, score standardization has been proposed to compare systems across collections and minimize the effect of outlier runs on specific topics. The underlying idea is to account for the difficulty of topics, so that systems are scored relative to it. Webber et al. first proposed standardization through a non-linear transformation with the standard normal distribution, and recently Sakai proposed a simple linear transformation. In this paper, we show that both approaches are actually special cases of a simple standardization which assumes specific distributions for the per-topic scores. From this viewpoint, we argue that a transformation based on the empirical distribution is the most appropriate choice for this kind of standardization. Through a series of experiments on TREC data, we show the benefits of our proposal in terms of score stability and statistical test behavior.

Statistical significance,
Student’s t-test
Wilcoxon test
Sign test
Bootstrap
Permutation
Simulation
Type I and Type II errors

http://resolver.tudelft.nl/uuid:d2ee6a8f-b79d-41b0-9731-1bdc333c831f

https://doi.org/10.1145/3331184.3331315

ACM DL, New York, USA

978-1-4503-6172-9

SIGIR'19 Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

SIGIR 2019: the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2019-07-21 → 2019-07-25, Cité des Sciences, Paris, France

Institutional Repository

conference paper

© 2019 Julián Urbano, H.A. De Lima, A. Hanjalic