User/Query-wise Metric Bounding in Learning to Rank

Abstract

Learning to Rank is the application of Machine Learning in order to create and optimize ranking functions. Most Learning to Rank methods follow a listwise approach and optimize a listwise loss function which closely resembles the same metric used in the evaluation. Popular listwise loss functions such as nDCG, AP and nRBP do not have consistent bounds across topics and do not account for instance-difficulty. As a result, the loss score does not solely reflect the performance of the model but also depends on the instance properties. During training, each instance is assumed to be equally informative, while in reality, this informativeness might depend on the difficulty of the instance. In this thesis, we propose four bounding methods which utilize some notion of instance-difficulty to produce difficulty-aware losses. Experimental results showed that, in most cases, optimizing a bounded variant of nDCG, AP or nRBP results in a consistent but marginal increase in the overall performance. More interestingly, we found that optimizing a bounded variant of nRBP and AP may increase the nDCG@k score, increasing the recommendation utility. Overall, our results show promising results for user/query-wise metric bounding in Learning to Rank, especially when applied to nRBP.