Print Email Facebook Twitter Exploring Copula-Based Models for the Stochastic Simulation of Information Retrieval Evaluation Data Title Exploring Copula-Based Models for the Stochastic Simulation of Information Retrieval Evaluation Data Author Theodorakopoulos, Dimitris (TU Delft Electrical Engineering, Mathematics and Computer Science) Contributor Urbano, Julián (mentor) Degree granting institution Delft University of Technology Programme Computer Science Date 2022-12-19 Abstract In the field of Information Retrieval (IR), the reliable evaluation of systems is a key component in order to progress the state-of-the-art. Much of IR research focuses on optimizing the various aspects of evaluation. Stochastic simulation is one technique that can be used to assist this kind of research. It allows researchers to overcome certain limitations associated with IR data, such as limited size, and lack of control. Recently, there have been two parallel lines of work that use stochastic simulation to study the question of "which statistical significance test is optimal for IR evaluation data?". Surprisingly, the authors reach different conclusions, despite the fact that both use stochastic simulation. One line of work, lead by Urbano et al., simulates scores for a fixed set of systems on new random topics, and concluded that the t-test is optimal. Another line of work, lead by Parapar et al., simulates new random retrieval runs for a fixed set of topics, and concluded that the Wilcoxon test is optimal. Interestingly these two tests are the most popular in IR literature. In an attempt to shed some light on this disagreement between the two conclusions, we made a first attempt at providing some empirical evidence regarding the quality of the simulation approach that was used by Urbano et al. Our main findings is that the quality of the simulation is moderately good, and also discovered some opportunities to refine it. In addition, we proposed a new model selection criterion, that showed some promising results, and in many cases managed to select models more optimally than other, more established criteria, such as AIC. Subject Information RetrievalEvaluationCopulaSimulation To reference this document use: http://resolver.tudelft.nl/uuid:e72a05ef-df32-4c08-aa1d-95d8c5828a2a Part of collection Student theses Document type master thesis Rights © 2022 Dimitris Theodorakopoulos Files PDF msc_thesis_4620534.pdf 1.73 MB Close viewer /islandora/object/uuid:e72a05ef-df32-4c08-aa1d-95d8c5828a2a/datastream/OBJ/view