Exploring Copula-Based Models for the Stochastic Simulation of Information Retrieval Evaluation Data

In the field of Information Retrieval (IR), the reliable evaluation of systems is a key component in order to progress the state-of-the-art. Much of IR research focuses on optimizing the various aspects of evaluation. Stochastic simulation is one technique that can be used to assist this kind of research. It allows researchers to overcome...

User/Query-wise Metric Bounding in Learning to Rank

Learning to Rank is the application of Machine Learning in order to create and optimize ranking functions. Most Learning to Rank methods follow a listwise approach and optimize a listwise loss function which closely resembles the same metric used in the evaluation. Popular listwise loss functions such as nDCG, AP and nRBP do not have consistent...

Understanding Context Effects in the Evaluation of Music Similarity

This work analyses context effect in the evaluation of music similarity performed by human annotators to better understand the impact of context effects in the current annotation protocol of Music Information Retrieval Evaluation eXchange (MIREX). Human annotators are known to be subjective when giving similarity judgements. The Audio Music...

Dealing with overconfidence and bias in low-cost evaluation of audio music similarity

The music information retrieval community has long recognized the need for better evaluation. Frameworks as comprehensive as in the field of text retrieval are missing. An important problem of evaluation for music retrieval is the high cost. This work continues on the proposed low-cost evaluation model. The low-cost model allows for cheap...