Neither the t-test nor Welch’s test

Abstract

Student’s t-test and Welch’s t-test are the most common defaults for comparing two independent samples, but each rests on often-violated assumptions. We compared the two tests in simulations varying sample-size imbalance, variances, and skewness, plus an empirical sampling study of gender differences on two psychological scales; further simulations compared both classical tests with the Yuen-Welch test, the t-test on ranks, Welch’s t-test on ranks, a permutation-based Welch’s test, the Brunner-Munzel test, the Kolmogorov–Smirnov test, and the Anderson–Darling test. Under unequal sample sizes, the t-test failed to maintain nominal Type I error when standard deviations also differed (the classical Welch motivation), while Welch’s test inflated Type I error when distributions were skewed, with the false positive rate reaching approximately 6%, 7.5%, and 9% at population skewness 1, 2, and 3 (nominal 5%). When sample sizes were equal, both classical tests held nominal Type I error; under unequal sample sizes with skewness, both lost power to robust alternatives, so the t-test was no remedy for Welch’s inflation. Among the alternatives, the permutation-based Welch’s test held the nominal Type I error across the factorial design while preserving the original measurement scale, making it a defensible default in the present simulations when the research question concerns equality of means. The Anderson–Darling test attained relatively high power when the two skewed populations differed simultaneously in mean and variability, and is a strong candidate when the research question concerns whether two distributions differ rather than whether their means differ specifically.