SimIIR 3
A Framework for the Simulation of Interactive and Conversational Information Retrieval
Leif Azzopardi (University of Strathclyde)
Timo Breuer (Cologne University of Applied Sciences)
Björn Engelmann (Cologne University of Applied Sciences)
Christin Kreutz (University of Applied Sciences Mittelhessen)
Sean MacAvaney (University of Glasgow)
David Maxwell (TU Delft - Web Information Systems)
Andrew Parry (University of Glasgow)
Adam Roegiest (Triangle Lab)
Xi Wang (University of Sheffield)
Saber Zerhoudi (University of Passau)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Evaluating the interactions between users and systems presents many challenges. Simulation offers a reliable, re-usable, and repeatable methodology to explore how different users, user behaviours and/or retrieval systems impact performance. With Large Language Models and Generative AI now widely available and accessible, new affordances are possible. These allow researchers to create more ''realistic'' simulated users that can generate queries and judge items like humans, and to develop new retrieval systems where responses and interactions are conversational and based on retrieval augmented generation. This resource paper presents a community-led initiative to update the Simulation of Interactive Information Retrieval (SimIIR) Framework to enable the simulation of conversational search using LLMs. The largest update provides a conversational search workflow which involves a number of new possible interactions with a search system or agent - enabling a host of new development and evaluation opportunities. Other developments include the Markovian Users, Cognitive States, LLM-based components for assessing snippets/documents/responses, generating queries, deciding on when to stop/continue, and PyTerrier integration. This paper aims to mark the release of SimIIR 3.0 and invites the community to build, extend, and use the resource.