Designing and Diagnosing Models for Conversational Search and Recommendation

Designing and Diagnosing Models for Conversational Search and Recommendation

Penha, G. (TU Delft Web Information Systems)

Houben, G.J.P.M. (promotor)
Hauff, C. (promotor)

Delft University of Technology

2023-05-24

Conversational search is a sub-field of Information Retrieval (IR) that focuses on solving information needs through natural language conversations. Searching for information is an inherently interactive task, and conversations offer a promising solution. One that might change the current search paradigm. In this thesis, we focus on retrieval and ranking approaches for conversational search systems, which are core IR technologies that have been progressing for decades.

First, we contribute with resources we created and which are used throughout the thesis. Namely, we introduce a novel dataset of information-seeking dialogues: MANtIS, as well as a library to train and evaluate models for the task of conversation response ranking: transformer-rankers.

Considering a two-stage pipeline for conversational search, we propose approaches for retrieval and also for re-ranking responses. We start by empirically comparing sparse and dense approaches for the first-stage retrieval of responses for dialogues. Next, we go to the second stage of the pipeline and use notions of difficulty to improve response re-rankers. We start with a curriculum learning approach that starts with easy dialogues and moves progressively to harder ones during training. We also investigate how difficult a dialogue can be when predicting the relevance of responses, by proposing models which allow for estimating their uncertainty.

Finally, we move on to evaluating what is the behavior and limitations of retrieval and ranking models for conversational search. We start by evaluating what is the effect of categories of language variations of queries in retrieval pipelines. Additionally, we evaluate what are the capabilities of heavily pre-trained language models for different conversational recommendation tasks.

With this thesis, we make scientific contributions to the field by providing resources, improving retrieval and re-rankers, and enabling a better understanding of models. We hope our contributions can be used as a foundation for future work in conversational search, enabling agents that can improve information-seeking interactions.

conversational search
ranking models
model understanding

https://doi.org/10.4233/uuid:acdfb704-6310-4b28-b884-4bd3e78b3f84

SIKS Dissertation Series, 2023-18

Institutional Repository

doctoral thesis

© 2023 G. Penha