- document
-
Bauer, Christine (author), Carterette, Ben (author), Ferro, Nicola (author), Fuhr, Norbert (author), Beel, Joeran (author), Breuer, Timo (author), Clarke, Charles L. A. (author), Dietz, Laura (author), Urbano, Julián (author)This report documents the program and the outcomes of Dagstuhl Seminar 23031 "Frontiers of Information Access Experimentation for Research and Education", which brought together 38 participants from 12 countries. The seminar addressed technology-enhanced information access (information retrieval, recommender systems, natural language processing)...journal article 2023
- document
-
Li, Roger Zhe (author), Urbano, Julián (author), Hanjalic, A. (author)Mainstream bias, where some users receive poor recommendations because their preferences are uncommon or simply because they are less active, is an important aspect to consider regarding fairness in recommender systems. Existing methods to mitigate mainstream bias do not explicitly model the importance of these non-mainstream users or, when...conference paper 2023
- document
-
Urbano, Julián (author), Corsi, M. (author), Hanjalic, A. (author)Statistical significance tests are the main tool that IR practitioners use to determine the reliability of their experimental evaluation results. The question of which test behaves best with IR evaluation data has been around for decades, and has seen all kinds of results and recommendations. Definitive answer to this question has recently...conference paper 2021
- document
-
Li, Roger Zhe (author), Urbano, Julián (author), Hanjalic, A. (author)In a collaborative-filtering recommendation scenario, biases in the data will likely propagate in the learned recommendations. In this paper we focus on the so-called mainstream bias: the tendency of a recommender system to provide better recommendations to users who have a mainstream taste, as opposed to non-mainstream users. We propose NAECF,...conference paper 2021
- document
-
Li, Roger Zhe (author), Urbano, Julián (author), Hanjalic, A. (author)Direct optimization of IR metrics has often been adopted as an approach to devise and develop ranking-based recommender systems. Most methods following this approach (e.g. TFMAP, CLiMF, Top-N-Rank) aim at optimizing the same metric being used for evaluation, under the assumption that this will lead to the best performance. A number of studies...conference paper 2021
- document
-
Schreiber, Hendrik (author), Urbano, Julián (author), Müller, Meinard (author)With the advent of deep learning, global tempo estimation accuracy has reached a new peak, which presents a great opportunity to evaluate our evaluation practices. In this article, we discuss presumed and actual applications, the pros and cons of commonly used metrics, and the suitability of popular datasets. To guide future research, we present...journal article 2020
- document
-
Roitero, Kevin (author), Brunello, Andrea (author), Urbano, Julián (author), Mizzaro, Stefano (author)Recently proposed methods allow the generation of simulated scores representing the values of an effectiveness metric, but they do not investigate the generation of the actual lists of retrieved documents. In this paper we address this limitation: we present an approach that exploits an evolutionary algorithm and, given a metric score,...conference paper 2019
- document
-
Bogdanov, Dmitry (author), Porter, Alastair (author), Schreiber, Hendrik (author), Urbano, Julián (author), Oramas, Sergio (author)This paper introduces the AcousticBrainz Genre Dataset, a large-scale collection of hierarchical multi-label genre annotations from different metadata sources. It allows researchers to explore how the same music pieces are annotated differently by different communities following their<br/>own genre taxonomies, and how this could be addressed<br/...conference paper 2019
- document
-
Bogdanov, Dmitry (author), Porter, Alastair (author), Schreiber, Hendrik (author), Urbano, Julián (author), Oramas, Sergio (author)This paper introduces the AcousticBrainz Genre Dataset, a large-scale collection of hierarchical multi-label genre annotations from different metadata sources. It allows researchers to explore how the same music pieces are annotated differently by different communities following their own genre taxonomies, and how this could be addressed by...conference paper 2019
- document
-
Kim, Jaehun (author), Urbano, Julián (author), Liem, C.C.S. (author), Hanjalic, A. (author)Inspired by the success of deploying deep learning in the fields of Computer Vision and Natural Language Processing, this learning paradigm has also found its way into the field of Music Information Retrieval. In order to benefit from deep learning in an effective, but also efficient manner, deep transfer learning has become a common approach...journal article 2019
- document
-
Sarasua, Alvaro (author), Urbano, Julián (author), Gómez, Emilia (author)Metaphors are commonly used in interface design within Human-Computer Interaction (HCI). Interface metaphors provide users with a way to interact with the computer that resembles a known activity, giving instantaneous knowledge or intuition about how the interaction works. A widely used one in Digital Musical Instruments (DMIs) is the conductor...journal article 2019
- document
-
Urbano, Julián (author), De Lima, H.A. (author), Hanjalic, A. (author)Statistical significance testing is widely accepted as a means to assess how well a difference in effectiveness reflects an actual difference between systems, as opposed to random noise because of the selection of topics. According to recent surveys on SIGIR, CIKM, ECIR and TOIS papers, the t-test is the most popular choice among IR researchers....conference paper 2019
- document
-
Urbano, Julián (author), De Lima, H.A. (author), Hanjalic, A. (author)In test collection based evaluation of IR systems, score standardization has been proposed to compare systems across collections and minimize the effect of outlier runs on specific topics. The underlying idea is to account for the difficulty of topics, so that systems are scored relative to it. Webber et al. first proposed standardization...conference paper 2019
- document
-
Kim, Jaehun (author), Urbano, Julián (author), Liem, C.C.S. (author), Hanjalic, A. (author)Deep neural networks have frequently been used to directly learn representations useful for a given task from raw input data. In terms of overall performance metrics, machine learning solutions employing deep representations frequently have been reported to greatly outperform those using hand-crafted feature representations. At the same time,...journal article 2019
- document
-
Bogdanov, Dmitry (author), Porter, Alastair (author), Urbano, Julián (author), Schreiber, Hendrik (author)This paper provides an overview of the AcousticBrainz Genre Task organized as part of the MediaEval 2018 Benchmarking Initiative for Multimedia Evaluation. The task is focused on content-based music genre recognition using genre annotations from multiple sources and large-scale music features data available in the AcousticBrainz database. The...conference paper 2018
- document
-
Urbano, Julián (author), Flexer, Arthur (author)Nearly since the beginning, the ISMIR and MIREX communities have promoted rigor in experimentation through the creation of datasets and the practice of statistical hypothesis testing to determine the reliability of the improvements observed with those datasets. In fact, MIR researchers have adopted a certain way of going about statistical...abstract 2018
- document
-
Marrero Llinares, M. (author), Urbano, Julián (author)Named Entity Recognition is a basic task in Information Extraction that aims at identifying entities of interest within full text documents. The patterns used to recognize entities can be rule based, as in the popular JAPE system. However, hand-crafting effective patterns is often difficult, and yet there is little research devoted to methods...journal article 2018
- document
-
Urbano, Julián (author), Nagler, Thomas (author)Part of Information Retrieval evaluation research is limited by the fact that we do not know the distributions of system effectiveness over the populations of topics and, by extension, their true mean scores. The workaround usually consists in resampling topics from an existing collection and approximating the statistics of interest with the...conference paper 2018
- document
-
Bogdanov, Dmitry (author), Porter, Alastair (author), Urbano, Julián (author), Schreiber, Hendrik (author)This paper provides an overview of the AcousticBrainz Genre Task <p style="margin: 0cm 0cm 0pt; line-height: normal;">organized as part of the MediaEval 2017 Benchmarking Initiative for Multimedia Evaluation. The task is focused on content-based music genre recognition using genre annotations from multiple sources and large-scale music features...conference paper 2017
- document
-
Urbano, Julián (author), Marrero Llinares, M. (author)The Kendall tau and AP correlation coefficients are very commonly use to compare two rankings over the same set of items. Even though Kendall tau was originally defined assuming that there are no ties in the rankings, two alternative versions were soon developed to account for ties in two different scenarios: measure the accuracy of an...conference paper 2017