Offline evaluation of Information Retrieval systems with its benefits of being overall cheaper than online evaluation still suffers from data sparsity. It has been established that LLMs can be used for labeling relevance much cheaper and faster than human annotators, while providing similar results. However it has not been measured how those labels affect the alignment between offline and online evaluation. In this work we present a labeling pipeline for a subset of queries from MS Marco. We then compare the labels against the baseline of Rank-DistiLLM dataset. We finalize by measuring alignment against external online metrics. In the results we observe that the metrics show strong correlation to both TREC 2019 and Rank-DistiLLM baselines. Measured alignment to Online metrics is moderate; However negative correlation present requires additional analysis.

https://huggingface.co/datasets/SemiGlassFace/CSE3000 ... Read more