Extending Rank-Biased Overlap (RBO) to Relevance Profiles
T.H.J. Houben (TU Delft - Electrical Engineering, Mathematics and Computer Science)
E.A. Markatou – Graduation committee member (TU Delft - Cyber Security)
Julián Urbano – Mentor (TU Delft - Multimedia Computing)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Rank-Biased Overlap (RBO) is a widely used metric for comparing ranked lists, due to its ability to handle incomplete and non-conjoint rankings while emphasizing top-ranked items. However, traditional RBO only considers the identity of ranked items, ignoring any associated relevance values. In many real-world applications, different systems may retrieve non-overlapping documents with similar informational value. This paper proposes an extension of RBO that incorporates graded relevance scores, enabling the comparison of rankings based on the information they convey rather than shared items alone.
Two relevance-aware variants for redefining RBO are proposed using cumulative gain.
These variants are evaluated and analyzed using TREC ad hoc and simulated data, comparing them with each other and against standard RBO. The results demonstrate that the new RBO variants provide a more informative similarity measure when comparing rankings with differing identities but similar relevance patterns.