Extending rank correlation coefficients to relevance profiles
A. Vezzuto (TU Delft - Electrical Engineering, Mathematics and Computer Science)
Julián Urbano – Mentor (TU Delft - Multimedia Computing)
E.A. Markatou – Graduation committee member (TU Delft - Cyber Security)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Frequently used in modern applications, rankings provide users with a list of the most relevant items. In information retrieval research, the τ, τap, and τh correlation coefficients are commonly applied to assess the similarity of the underlying systems by comparing the rankings they produce. Traditionally, these comparisons focus solely on item ranking, but introducing relevance values has enabled systems to be analysed based on how element utility relates to retrieval order. In this work, τ, τap, and τh are extended to incorporate relevance values, presenting several coefficients rooted in an axiomatic approach. These measures compare the utility of items or indices, enabling a granular relevance-based ranking comparison. Overall, the results demonstrate that including relevance judgments leads to significant deviations from traditional rank correlation metrics, highlighting the impact of relevance-aware measures in evaluating system performance and similarity.