Toward Rank Correlation as a Measure of Confidence in Information Retrieval Experiment Results
A.I. Bavdaz (TU Delft - Electrical Engineering, Mathematics and Computer Science)
Julián Urbano – Mentor
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
In the field of Information Retrieval (IR), test collections are an important part of IR system evaluation. When evaluating IR systems on a test collection, the results may not accurately represent the performance of the systems on topics not contained in that test collection. Therefore, we want to get a sense of the accuracy of results on a given test collection. In this thesis, we use an approach that estimates the accuracy of test collections by estimating rank correlation between the observed and true mean scores of systems. We further evaluate this approach on new data and develop interval estimators as well. This way we provide a better sense of confidence on the system evaluation results by accounting for the inherent variability in sampling topics.