Probing BERT for Ranking Abilities

Conference Paper (2023)
Author(s)

Jonas Wallat (L3S)

Fabian Beringer (L3S)

Abhijit Anand (L3S)

Avishek Anand (TU Delft - Web Information Systems, L3S)

Research Group
Web Information Systems
Copyright
© 2023 Jonas Wallat, Fabian Beringer, Abhijit Anand, A. Anand
DOI related publication
https://doi.org/10.1007/978-3-031-28238-6_17
More Info
expand_more
Publication Year
2023
Language
English
Copyright
© 2023 Jonas Wallat, Fabian Beringer, Abhijit Anand, A. Anand
Research Group
Web Information Systems
Bibliographical Note
Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.@en
Pages (from-to)
255-273
ISBN (print)
978-3-031-28237-9
ISBN (electronic)
978-3-031-28238-6
Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Abstract

Contextual models like BERT are highly effective in numerous text-ranking tasks. However, it is still unclear as to whether contextual models understand well-established notions of relevance that are central to IR. In this paper, we use probing, a recent approach used to analyze language models, to investigate the ranking abilities of BERT-based rankers. Most of the probing literature has focussed on linguistic and knowledge-aware capabilities of models or axiomatic analysis of ranking models. In this paper, we fill an important gap in the information retrieval literature by conducting a layer-wise probing analysis using four probes based on lexical matching, semantic similarity as well as linguistic properties like coreference resolution and named entity recognition. Our experiments show an interesting trend that BERT-rankers better encode ranking abilities at intermediate layers. Based on our observations, we train a ranking model by augmenting the ranking data with the probe data to show initial yet consistent performance improvements (The code is available at https://github.com/yolomeus/probing-search/ ).

Files

978_3_031_28238_6_17.pdf
(pdf | 0 Mb)
- Embargo expired in 17-09-2023
License info not available