Dealing with overconfidence and bias in low-cost evaluation of audio music similarity
Tim Buckers (TU Delft - Electrical Engineering, Mathematics and Computer Science)
Julian Urbano Merino – Mentor (TU Delft - Multimedia Computing)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
The music information retrieval community has long recognized the need for better evaluation. Frameworks as comprehensive as in the field of text retrieval are missing. An important problem of evaluation for music retrieval is the high cost. This work continues on the proposed low-cost evaluation model. The low-cost model allows for cheap evaluation with a degree of uncertainty. Alongside the evaluation results, confidence is indicated for interpretability. The confidence is however overconfident. This work breaks down the causes and introduces solutions that reduce this overconfidence. The evaluation results of certain metrics are structurally overestimated. The source of this problem is found and an appropriate solution is explained.