MC
M. Corsi
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
5 records found
1
Rank-biased Overlap (RBO) is a measure that is used to compare two rankings against each other mathematically using a hyperparameter for persistence, p, to define the importance of items higher up in the rankings. This is able to follow the properties of incompleteness, indefiniteness, and top-heaviness for its results, making it a flexible option for rank similarity. In traditional RBO, the intersection of the items in each of the rankings is weighted by the persistence to reach the final value for RBO as it tends towards infinity. RBO has several assumptions, such as on what happens when rankings are tied, having an infinitely long ranking, and a degree of conjointness between the rankings. In this paper, two new variations are derived on the aspects of having the rankings be fully conjoint, as well as the aspect of having a known finite domain for the rankings. These are described through the equations of RBOc, for fully conjoint rankings, and RBOf , for rankings within a known finite domain. While RBOc tends to be slightly larger and RBOf tends to be smaller when compared to traditional RBO, both can be more fitting depending on the greater context of their use cases.
...
Rank-biased Overlap (RBO) is a measure that is used to compare two rankings against each other mathematically using a hyperparameter for persistence, p, to define the importance of items higher up in the rankings. This is able to follow the properties of incompleteness, indefiniteness, and top-heaviness for its results, making it a flexible option for rank similarity. In traditional RBO, the intersection of the items in each of the rankings is weighted by the persistence to reach the final value for RBO as it tends towards infinity. RBO has several assumptions, such as on what happens when rankings are tied, having an infinitely long ranking, and a degree of conjointness between the rankings. In this paper, two new variations are derived on the aspects of having the rankings be fully conjoint, as well as the aspect of having a known finite domain for the rankings. These are described through the equations of RBOc, for fully conjoint rankings, and RBOf , for rankings within a known finite domain. While RBOc tends to be slightly larger and RBOf tends to be smaller when compared to traditional RBO, both can be more fitting depending on the greater context of their use cases.
As a point estimate of the similarity score between two possibly indefinite rankings, extrapolated rank-biased overlap (RBOEXT) uses the assumption that the agreement observed at the last evaluation depth continues indefinitely across the unseen tails of the two lists. This assumption does not account for any patterns that occur in the visible prefixes, imposing a strict restriction on the extrapolation. In an effort to improve the accuracy of RBOEXT, three reformulations with a relaxed theoretical basis are proposed in this paper: one continually re-uses the agreement from the previous depth while the other two rely on regression to fit a function on the seen agreements. Using synthetic data, the performance of these new extrapolation methods is compared to the original's in terms of closeness to the true RBO score as well as the average distance between assumed and actual agreement in the rankings' unseen tails. Overall, an impactful difference is observed in the estimates of agreement generated by the four approaches: as the trends from the visible prefixes are barely captured by the simpler techniques or closely-reproduced by the more flexible ones, the trade-off between under- and overfitting becomes increasingly relevant. The results thus indicate a need for some middle-ground to be established such that it factors in the observed patterns while also generalizing well for the tails.
...
As a point estimate of the similarity score between two possibly indefinite rankings, extrapolated rank-biased overlap (RBOEXT) uses the assumption that the agreement observed at the last evaluation depth continues indefinitely across the unseen tails of the two lists. This assumption does not account for any patterns that occur in the visible prefixes, imposing a strict restriction on the extrapolation. In an effort to improve the accuracy of RBOEXT, three reformulations with a relaxed theoretical basis are proposed in this paper: one continually re-uses the agreement from the previous depth while the other two rely on regression to fit a function on the seen agreements. Using synthetic data, the performance of these new extrapolation methods is compared to the original's in terms of closeness to the true RBO score as well as the average distance between assumed and actual agreement in the rankings' unseen tails. Overall, an impactful difference is observed in the estimates of agreement generated by the four approaches: as the trends from the visible prefixes are barely captured by the simpler techniques or closely-reproduced by the more flexible ones, the trade-off between under- and overfitting becomes increasingly relevant. The results thus indicate a need for some middle-ground to be established such that it factors in the observed patterns while also generalizing well for the tails.
Effects of the assumption on ties in unseen parts of a ranking
What will happen if we relax the assumption that ties do not occur in unseen parts?
Rankings are more present in our daily lives than most people realize. Whether you are browsing Netflix and getting movies or shows based on your previous likes or dislikes, or you want to compare search engine results. To use rankings in the field of Computer Science a rank similarity is needed. Rank-Biased Overlap is one of those. It is top-weighted, can be used on uneven rankings, and when only a part of the ranking is known. A well-known problem in rank similarity measures is ties. There have been some ways of dealing with ties proposed since RBO was introduced. These ways have been shown to be promising but they only relate to the seen part. The unseen part of rankings is still a new concept with little research done about it. This paper aims to change that a bit. First, a full explanation is given of the three variations of dealing with ties. Then using these variants we show how the assumption that no ties exist in the unseen part affects these variants. Also, the current extrapolation method is researched as there is also a big influence of the above-mentioned assumption. We then use simulated data to give a clear data visualization to show how the theory relates to practice. We have tried to be clear and concise with our explanations and data visualizations so future researchers can use this paper to improve and progress RBO in the world of rank similarity measures.
...
Rankings are more present in our daily lives than most people realize. Whether you are browsing Netflix and getting movies or shows based on your previous likes or dislikes, or you want to compare search engine results. To use rankings in the field of Computer Science a rank similarity is needed. Rank-Biased Overlap is one of those. It is top-weighted, can be used on uneven rankings, and when only a part of the ranking is known. A well-known problem in rank similarity measures is ties. There have been some ways of dealing with ties proposed since RBO was introduced. These ways have been shown to be promising but they only relate to the seen part. The unseen part of rankings is still a new concept with little research done about it. This paper aims to change that a bit. First, a full explanation is given of the three variations of dealing with ties. Then using these variants we show how the assumption that no ties exist in the unseen part affects these variants. Also, the current extrapolation method is researched as there is also a big influence of the above-mentioned assumption. We then use simulated data to give a clear data visualization to show how the theory relates to practice. We have tried to be clear and concise with our explanations and data visualizations so future researchers can use this paper to improve and progress RBO in the world of rank similarity measures.
Average Rank-Biased Overlap between independent rankings
Revealing average benchmarks: An Empirical Investigation
Rankings play a crucial role in various contexts but often exhibit incompleteness, top-weightedness, and indefiniteness. Comparing rankings can reveal underlying similarities, yet traditional correlation coefficients like Kendall's tau do not adequately address these complexities. Rank-Biased Overlap (RBO) addresses these challenges by accommodating differences in rank length, appropriately weighting ranks, and minimizing data assumptions. This paper investigates the average Rank-Biased Overlap (RBO) between independent rankings, addressing the need for clearly indicated reference values similar to those of correlation coefficients. Our study explores how the expected RBO changes with varying p-parameters, prefix lengths, and degrees of conjointness between domains.
To facilitate this analysis, an algorithm is developed that performs extensive simulations across different values of p, list and domain sizes. By analyzing the simulation results, trends are provided in the average RBO between independent rankings based on these varying parameters and establish relevant reference values. This study focuses on scenarios where prefixes are of the same length and there are no ties in the rankings. ...
To facilitate this analysis, an algorithm is developed that performs extensive simulations across different values of p, list and domain sizes. By analyzing the simulation results, trends are provided in the average RBO between independent rankings based on these varying parameters and establish relevant reference values. This study focuses on scenarios where prefixes are of the same length and there are no ties in the rankings. ...
Rankings play a crucial role in various contexts but often exhibit incompleteness, top-weightedness, and indefiniteness. Comparing rankings can reveal underlying similarities, yet traditional correlation coefficients like Kendall's tau do not adequately address these complexities. Rank-Biased Overlap (RBO) addresses these challenges by accommodating differences in rank length, appropriately weighting ranks, and minimizing data assumptions. This paper investigates the average Rank-Biased Overlap (RBO) between independent rankings, addressing the need for clearly indicated reference values similar to those of correlation coefficients. Our study explores how the expected RBO changes with varying p-parameters, prefix lengths, and degrees of conjointness between domains.
To facilitate this analysis, an algorithm is developed that performs extensive simulations across different values of p, list and domain sizes. By analyzing the simulation results, trends are provided in the average RBO between independent rankings based on these varying parameters and establish relevant reference values. This study focuses on scenarios where prefixes are of the same length and there are no ties in the rankings.
To facilitate this analysis, an algorithm is developed that performs extensive simulations across different values of p, list and domain sizes. By analyzing the simulation results, trends are provided in the average RBO between independent rankings based on these varying parameters and establish relevant reference values. This study focuses on scenarios where prefixes are of the same length and there are no ties in the rankings.
Adaptive Synthetic Generation of Indefinite Rankings
Enhancing Algorithm Flexibility with Tunable Conjointness, Overlap, and Tie Distribution
Reducing the similarity of two ranked lists to a single value proves to be useful in various fields of research and industry, such as Information Retrieval and Recommender Systems, leading to the introduction of several similarity measures. One such measure is Rank-Biased Overlap and its variants, possessing qualities such as the ability to handle incomplete rankings, non-conjoint ranking domains and the graceful evaluation of ranking pairs with tied items.
Comparing the performance of similarity measures, or comparing variants of a single measure, requires the presence of ranking data. In certain cases, generating synthetic ranking data may be a more viable option than using real data. However, a review of existing literature reveals a lack of parametrisable synthetic ranking algorithms. This paper introduces a novel method to generate a pair of rankings where one can tailor the conjointness of ranking domains, influence the ranking overlap as a function of depth and tune the presence of tie groups in a probabilistic manner. The paper demonstrates the output of the algorithm when varying the input parameters, verifying the methods performance empirically and statistically. ...
Comparing the performance of similarity measures, or comparing variants of a single measure, requires the presence of ranking data. In certain cases, generating synthetic ranking data may be a more viable option than using real data. However, a review of existing literature reveals a lack of parametrisable synthetic ranking algorithms. This paper introduces a novel method to generate a pair of rankings where one can tailor the conjointness of ranking domains, influence the ranking overlap as a function of depth and tune the presence of tie groups in a probabilistic manner. The paper demonstrates the output of the algorithm when varying the input parameters, verifying the methods performance empirically and statistically. ...
Reducing the similarity of two ranked lists to a single value proves to be useful in various fields of research and industry, such as Information Retrieval and Recommender Systems, leading to the introduction of several similarity measures. One such measure is Rank-Biased Overlap and its variants, possessing qualities such as the ability to handle incomplete rankings, non-conjoint ranking domains and the graceful evaluation of ranking pairs with tied items.
Comparing the performance of similarity measures, or comparing variants of a single measure, requires the presence of ranking data. In certain cases, generating synthetic ranking data may be a more viable option than using real data. However, a review of existing literature reveals a lack of parametrisable synthetic ranking algorithms. This paper introduces a novel method to generate a pair of rankings where one can tailor the conjointness of ranking domains, influence the ranking overlap as a function of depth and tune the presence of tie groups in a probabilistic manner. The paper demonstrates the output of the algorithm when varying the input parameters, verifying the methods performance empirically and statistically.
Comparing the performance of similarity measures, or comparing variants of a single measure, requires the presence of ranking data. In certain cases, generating synthetic ranking data may be a more viable option than using real data. However, a review of existing literature reveals a lack of parametrisable synthetic ranking algorithms. This paper introduces a novel method to generate a pair of rankings where one can tailor the conjointness of ranking domains, influence the ranking overlap as a function of depth and tune the presence of tie groups in a probabilistic manner. The paper demonstrates the output of the algorithm when varying the input parameters, verifying the methods performance empirically and statistically.