Non-Monotonicity in Empirical Learning Curves

Non-Monotonicity in Empirical Learning Curves: Identifying non-monotonicity through slope approximations on discrete points

Socol, Codrin (TU Delft Electrical Engineering, Mathematics and Computer Science)

Viering, T.J. (mentor)
Krijthe, J.H. (mentor)
Yue, Z. (graduation committee)

Delft University of Technology

Computer Science and Engineering

CSE3000 Research Project

2023-06-28

Learning curves are used to shape the performance of a Machine Learning (ML) model with respect to the size of the set used for training it. It was commonly thought that adding more training samples would increase the model's accuracy (i.e., they are monotone), but recent works show that may not always be the case. In other words, some learners on some problems show non-monotonic behaviour. To this extent, we introduce a new method to identify non-monotonicity in empirical learning curves by approximating the curve's slope through regression around the discrete points it is defined on.This paper formalises this metric and then evaluates its accuracy through different experiments. Finally, we run the proposed metric on a subset of the extensive Learning Curve Database (LCDB) by Mohr et al. to gain better insights into the problem of non-monotonicity of learning. We found that the metric can identify non-monotonicity in learning curves well (98% experimental accuracy) and does not consider small increases due to measurement error as non-monotonicity in the curve. Finally, we have identified that non-monotonicity may be a property of some classifiers, such as Linear Discriminant Analysis. Moreover, we identified that non-monotonicity is frequently observed in datasets with faster training times.

learning curve
non-monotonicity
meta-learning
LCDB
Machine Learning

http://resolver.tudelft.nl/uuid:3b7f24c8-08a9-4641-be82-38b880ac6898

Student theses

bachelor thesis

© 2023 Codrin Socol