How do Metric Score Distributions affect the Type i Error Rate of Statistical Significance Tests in Information Retrieval?

Statistical significance tests are the main tool that IR practitioners use to determine the reliability of their experimental evaluation results. The question of which test behaves best with IR evaluation data has been around for decades, and has seen all kinds of results and recommendations. Definitive answer to this question has recently...