ZZ
Z. Zhang
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
1 records found
1
To bridge the offline-online evaluation gap, we in- vestigate the use of large language models (LLMs) for scalable user simulation. While LLMs offer a promising approach, the reliability of such simu- lations is highly sensitive to prompt design. This study makes two main contributions: (1) we intro- duce a novel three-dimensional reliability frame- work for evaluating LLM-based user simulations, capturing stability, discrimination, and plausibil- ity; (2) we systematically compare three prompt- ing strategies—One-Shot, Multi-Shot, and Offline- Evaluation-informed—across 52 retrieval systems. Results show that Multi-Shot prompting achieves the best balance across all reliability dimensions, producing simulations that are both consistent and sensitive to system quality. In contrast, Offline- Evaluation-informed prompts align closely with of- fline metrics but fail to differentiate system perfor- mance. These findings provide practical guidance for designing reliable LLM-based user simulators, reducing reliance on costly online experiments.
...
To bridge the offline-online evaluation gap, we in- vestigate the use of large language models (LLMs) for scalable user simulation. While LLMs offer a promising approach, the reliability of such simu- lations is highly sensitive to prompt design. This study makes two main contributions: (1) we intro- duce a novel three-dimensional reliability frame- work for evaluating LLM-based user simulations, capturing stability, discrimination, and plausibil- ity; (2) we systematically compare three prompt- ing strategies—One-Shot, Multi-Shot, and Offline- Evaluation-informed—across 52 retrieval systems. Results show that Multi-Shot prompting achieves the best balance across all reliability dimensions, producing simulations that are both consistent and sensitive to system quality. In contrast, Offline- Evaluation-informed prompts align closely with of- fline metrics but fail to differentiate system perfor- mance. These findings provide practical guidance for designing reliable LLM-based user simulators, reducing reliance on costly online experiments.