Testing for no effect in regression problems
A permutation approach
Michal G. Ciszewski (TU Delft - Statistics)
J. Söhl (TU Delft - Statistics)
A. J.R. Leenen (Vrije Universiteit Amsterdam)
Bart van Trigt (TU Delft - Biomechanical Engineering, TU Delft - Biomechatronics & Human-Machine Control)
G Jongbloed (TU Delft - Statistics)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Often the question arises whether (Formula presented.) can be predicted based on (Formula presented.) using a certain model. Especially for highly flexible models such as neural networks one may ask whether a seemingly good prediction is actually better than fitting pure noise or whether it has to be attributed to the flexibility of the model. This paper proposes a rigorous permutation test to assess whether the prediction is better than the prediction of pure noise. The test avoids any sample splitting and is based instead on generating new pairings of (Formula presented.). It introduces a new formulation of the null hypothesis and rigorous justification for the test, which distinguishes it from the previous literature. The theoretical findings are applied both to simulated data and to sensor data of tennis serves in an experimental context. The simulation study underscores how the available information affects the test. It shows that the less informative the predictors, the lower the probability of rejecting the null hypothesis of fitting pure noise and emphasizes that detecting weaker dependence between variables requires a sufficient sample size.
help_outline