Frequently used in modern applications, rankings provide users with a list of the most relevant items. In information retrieval research, the τ, τap, and τh correlation coefficients are commonly applied to assess the similarity of the underlying systems by comparing the rankings
...
Frequently used in modern applications, rankings provide users with a list of the most relevant items. In information retrieval research, the τ, τap, and τh correlation coefficients are commonly applied to assess the similarity of the underlying systems by comparing the rankings they produce. Traditionally, these comparisons focus solely on item ranking, but introducing relevance values has enabled systems to be analysed based on how element utility relates to retrieval order. In this work, τ, τap, and τh are extended to incorporate relevance values, presenting several coefficients rooted in an axiomatic approach. These measures compare the utility of items or indices, enabling a granular relevance-based ranking comparison. Overall, the results demonstrate that including relevance judgments leads to significant deviations from traditional rank correlation metrics, highlighting the impact of relevance-aware measures in evaluating system performance and similarity.