Learning to Curve

Abstract

Learning curves in machine learning are useful tools to steer data acquisition and early stopping, representing the relation between training data quantity and performance. In existing literature, prediction of these learning curves largely relies on extrapolating from an initial set of observed performances and does not consider the characteristics of the underlying dataset, dubbed "metafeatures". Existing research of the utility of these metafeatures in the domain of learning curve prediction is sparse. Further, when it comes to measuring the fitness of predictions, the vast majority of literature uses Mean Squared Error. Different loss functions are seldom considered, and comparisons between different loss functions are not made.

In this Master Thesis we showcase metafeature-driven learning curve prediction by training two metalearners on a synthetic database of learning curves. We show that these metalearners strongly outperform the problem-average curve, which is indicative of meaningful task learning. The learning curves predictions produced by these metalearners perform comparable to curves obtained through direct curve fitting methods.

These results are obtained using 33 metafeatures. Four metafeature categories, comprising 12 metafeatures, are newly introduced in this research. Of these categories, we show high utility for three: Bayes Error, Negentropy and Surrogate Bayes. Of these, the Bayes Error shows highest utility. The best performing category of metafeature was the landmarkers, reinforcing their usage in literature.

We further show that the effect of chosen loss function on the shape of directly fitted curves on learning curve data is present but minor. The most notable difference appears in the exponent, where using a squared loss function yields a slightly lower (negative) exponent for power laws compared to the Mean Absolute Error. These trends do not readily appear to affect the shape of the resulting learning curve.

Evaluation or real-world data using the CC18-Database yielded no strong performance, suggesting that the synthetic learning curve database is not reflective of real problem datasets. Further research is needed to ascertain the nature of this discrepancy.