Print Email Facebook Twitter A New Perspective on Score Standardization Title A New Perspective on Score Standardization Author Urbano, Julián (TU Delft Multimedia Computing) De Lima, H.A. (TU Delft Multimedia Computing) Hanjalic, A. (TU Delft Intelligent Systems) Department Intelligent Systems Date 2019 Abstract In test collection based evaluation of IR systems, score standardization has been proposed to compare systems across collections and minimize the effect of outlier runs on specific topics. The underlying idea is to account for the difficulty of topics, so that systems are scored relative to it. Webber et al. first proposed standardization through a non-linear transformation with the standard normal distribution, and recently Sakai proposed a simple linear transformation. In this paper, we show that both approaches are actually special cases of a simple standardization which assumes specific distributions for the per-topic scores. From this viewpoint, we argue that a transformation based on the empirical distribution is the most appropriate choice for this kind of standardization. Through a series of experiments on TREC data, we show the benefits of our proposal in terms of score stability and statistical test behavior. Subject Statistical significance,Student’s t-testWilcoxon testSign testBootstrapPermutationSimulationType I and Type II errors To reference this document use: http://resolver.tudelft.nl/uuid:d2ee6a8f-b79d-41b0-9731-1bdc333c831f DOI https://doi.org/10.1145/3331184.3331315 Publisher ACM DL, New York, USA ISBN 978-1-4503-6172-9 Source SIGIR'19 Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval Event SIGIR 2019: the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2019-07-21 → 2019-07-25, Cité des Sciences, Paris, France Part of collection Institutional Repository Document type conference paper Rights © 2019 Julián Urbano, H.A. De Lima, A. Hanjalic Files PDF 077_new_perspective_score ... zation.pdf 1.56 MB Close viewer /islandora/object/uuid:d2ee6a8f-b79d-41b0-9731-1bdc333c831f/datastream/OBJ/view