Rank-Biased Overlap (RBO) is a widely used metric for comparing ranked lists, due to its ability to handle incomplete and non-conjoint rankings while emphasizing top-ranked items. However, traditional RBO only considers the identity of ranked items, ignoring any associated relevance values. In many real-world applications, different systems may retrieve non-overlapping documents with similar informational value. This paper proposes an extension of RBO that incorporates graded relevance scores, enabling the comparison of rankings based on the information they convey rather than shared items alone.

Two relevance-aware variants for redefining RBO are proposed using cumulative gain.
These variants are evaluated and analyzed using TREC ad hoc and simulated data, comparing them with each other and against standard RBO. The results demonstrate that the new RBO variants provide a more informative similarity measure when comparing rankings with differing identities but similar relevance patterns.