In Search of Best Learning Curve Model

Abstract

Learning curves have been used extensively to analyse learners' behaviour and practical tasks such as model selection, speeding up training and tuning models. Nonetheless, we still have a relatively limited understanding of the behaviour of learning curves themselves, in particular, whether there exists a parametric function that can best model all learning curves. Therefore, this study aims to determine which parametric models proposed over the years provide the best fit when applied to empirical learning curves. To answer this question, the study focuses on supervised learning and is divided into two parts: classification and regression tasks, and the learning curve data for each task was fitted using the Levenberg-Marquardt algorithm. Subsequently, the fitted models were analysed using the Friedman test, the Wilcoxon signed-rank test, and other metrics. The results indicate that a power law applies in most cases. However, a universal model has not been found, as the best model differs between classification and regression tasks, even though they belong to the power law family. Moreover, there are some deviations from these aggregate results when examining the learners individually, suggesting that a more granular approach is better suited for practical applications.