Motivation: Clustering is an unsupervised learning task with broad applications. Traditional clustering methods often rely on point estimates of model parameters, which can limit their ability to capture uncertainty. Bayesian clustering addresses this by incorporating uncertainty into parameter estimation. However, existing Bayesian inference methods like Markov Chain Monte Carlo and Variational Inference are computationally intensive and can produce biased approximations. To overcome these limitations, we propose Cluster-PFN, a transformer-based model inspired by Prior-Data Fitted Networks. Cluster-PFN simultaneously approximates the posterior distributions over the cluster assignments of individual data points and the total number of clusters for the given dataset in a single forward pass. Our model provides fast and accurate Bayesian clustering, supporting data with up to five features and conditioning on user-specified cluster counts.

Results: Our results demonstrate that Cluster-PFN can predict the number of clusters up to 20% more accurately than standard heuristics. It also outperforms the Bayesian Gaussian Mixture Model using Variational Inference (Bayesian GMM VI), achieving up to 60% higher scores on certain external metrics while being up to 20 times faster during inference. Additionally, Cluster-PFN surpasses both the traditional Gaussian Mixture Model and K-means++ across the same external evaluation metrics.

Learning curve extrapolation helps practitioners predict model performance at larger data scales, enabling better planning for data collection and computational resource allocation. This paper investigates when neural networks outperform parametric models for this task. We conduct a comprehensive comparison of LC-PFNs (Learning Curve Prior-Fitted Networks) and three established parametric models (POW4, MMF4, WBL4) using LCDB v1.1, a large-scale dataset with learning curves generated across 265 classification tasks and 24 learners. Surprisingly, we find that parametric models — especially POW4 and MMF4 — consistently outperform LC-PFN across all generalization scenarios and most cutoff regions. However, LC-PFN demonstrates competitive performance when extrapolating from early-stage data, ranking second-best at 10%, 30%, and 50% cutoffs. This suggests LC-PFNs can be valuable when only a small fraction of the learning curve is available. LC-PFN is particularly challenged by smooth and flat curves, but shows slightly improved performance on irregular patterns such as peaking and dipping curves, though it remains outperformed by all parametric models. These trends highlight a misalignment between LC-PFN’s training distribution and the real-world diversity of learning curves. Our findings emphasize the strength of parametric models under realistic conditions and suggest avenues for improving LC-PFNs through architectural flexibility and curve length variability during training.

One of the problems in continual learning, where models are trained sequentially on tasks, is a sudden drop in performance after switching to a new task, called stability gap. The presence of stability gap likely indicates that training is not done optimally. In this work we aim to address stability gap problem by using sharpness-aware optimization that biases convergence to flat minima. While flat minima are known to mitigate forgetting, their role in ensuring stable learning during task transitions remains unexplored. Through systematic analysis of two Entropy-SGD and C-Flat, we demonstrate that sharpness-aware optimizers produce smoother learning trajectories with reduced instability after task switch. Furthermore, we show that C-Flat’s second-order curvature approximation provides additional stabilization, suggesting that efficient Hessian-aware methods offer advantages for continual learning. The source code is available at Stability-Gap-SAM.

Learning curves represent the relationship between the amount of training data and the error rate in machine learning. An important use case for learning curves is extrapolating them in order to predict how much data is needed to achieve a certain performance. One way to do such extrapolations is using Deep Learning with a Prior-Fitted Network(PFN). This paper explores how training the PFN on an imbalanced dataset, i.e. containing learning curves from two or more machine learning models with a skewed distribution, affects the performance of the network. Research into imbalanced learning has shown that machine learning models can favor the more prevalent classes or data. Therefore, it is worthwhile to explore whether such trends can occur for the neural networks that we train for learning curve extrapolation. Our experiments focused on analyzing different imbalance scenarios and comparing them. Our results show that mixing learning curves from different learners can improve extrapolation performance in some cases, but the effect strongly depends on the learner characteristics and training proportions.

Accurately predicting a machine learning model’s final performance based on only partial training data can save substantial computational resources and guide early stopping, model selection, and automated machine learning (AutoML) workflows. Learning Curve Prior-Fitted Networks (LC-PFNs) are a recent data-driven approach to this problem, leveraging transformers trained on prior learning curves to make extrapolations. However, real-world training logs are often noisy and irregular—conditions under which the reliability of LC-PFNs remains largely untested. This thesis presents a systematic investigation into the robustness of LC-PFNs when exposed to noisy input data. Using LCDB 1.1—a large-scale dataset—we simulate various levels of noise by corrupting the input data with Gaussian perturbations and quantify how prediction accuracy degrades. To improve resilience, we study and propose two complementary mitigation strategies. The first injects artificial noise into the training data itself, either at a constant level or increasing gradually throughout training, encouraging the model to generalize across a spectrum of noise conditions. The second applies post-processing techniques at inference time, such as smoothing the input sequence with an exponential moving average or averaging multiple stochastic predictions using dropout. Our results show that standard LC-PFNs, trained only on clean data, are highly sensitive to even minor corruptions. In contrast, models trained with gradual noise exposure and evaluated with input smoothing achieve much greater robustness—reducing error by up to 75% under severe noise—while maintaining high accuracy in noise-free settings. This work demonstrates that substantial improvements in learning-curve extrapolation can be achieved without modifying model architecture, using general purpose techniques suitable for real-world deployment.

In the context of continual learning, recent work has identified a significant and recurring perfor- mance drop, followed by a gradual recovery, upon the introduction of a new task. This phenomenon is referred to as the stability gap. Investigating it and the potential solutions is essential, as such findings can reduce both the energy consumption and computational time required to prepare a high- performing agent. Given the strong influence of training procedures on model performance and sta- bility, we analyze how various optimizers –SGD, NAG, AdaGrad, RMSprop, Adam– and momentum values affect the stability gap. We expose a deep neural network to a sequence of digit-identification tasks with varying rotations, and track several met- rics to capture the components of the stability gap and the overall performance. Our results reveal that increasing momentum amplifies the steepness and depth of the gap, while shortening its duration. Within this simplified setup, RMSprop proves most effective in reducing the magnitude and duration of the drop while maintaining high overall perfor- mance.

In machine learning, learning curves are a metric that plots performance versus training set size. They inform decisions about data acquisition, model selection, and hyperparameter tuning. Despite their importance, recent research suggests that our understanding of learning curve behavior remains limited. In this work, we investigate learning curves from a classification perspective to better understand their structural properties. By framing learning curves as time series and applying time series classification (TSC) techniques, we uncover several key findings: (1) training accuracy curves are significantly more distinguishable across models than validation or test curves; (2) learning curves become more informative and discriminative after a sufficient number of anchor points; and (3) TSC models that emphasize global structural features outperform those focused on local or pointwise characteristics. These results not only offer new insights into the nature of learning curves but also suggest promising directions for future work, including the development of specialized models that move beyond conventional time series assumptions.

Domain shift is when the distribution of data differs between the training of a model and its testing. This can happen when the conditions of training are slightly different from the conditions that will happen when a model is tested or used. This is a problem for generalizability of a model. Learning curves are widely used in machine learning to predict how much data is needed when training a model. This paper will explore how domain shift impacts learning curve extrapolation using Learning Curve Prior Fitted Networks. We will explore the effect of domain shift on the performance of models while comparing different learners and groups of learners, thereby showing that domain shift is relevant to learning curve extrapolation and has a statistically significant impact on the accuracy of such extrapolations. We will also discuss how patterns like well-behavedness have an impact on this effect of domain shift, while also showing that is it not the full solution to predicting the effect.

Continual learning aims to train models that can incrementally acquire new knowledge over a sequence of tasks while retaining previously learned information, even in the absence of access to past data. A key challenge in this setting is maintaining stability at task transitions, where even methods like experience replay can suffer from temporary performance degradation known as the stability gap. In this work, we evaluate Layerwise Proximal Replay (LPR), a recently proposed optimisation strategy that constrains updates at the layer level to preserve internal representations of past data. We implement LPR on a simple multi-layer perceptron and benchmark it against an incremental joint training baseline on a domain-incremental variant of Rotated MNIST. To quantify the stability gap, we track accuracy drops immediately following task switches and compute local minima after transitions. Our results show that LPR consistently reduces the stability gap across a range of learning rates, with statistically significant improvements at higher values. However, this improvement comes at the cost of reduced performance on later tasks. These findings demonstrate that LPR significantly mitigates short-term performance degradation at task boundaries while maintaining high learning rates, offering a practical solution for increased stability in continual learning.

Task-based continual learning setups suffer from temporary dips in performance shortly after switching to new tasks, a phenomenon referred to as stability gap. State-of-the-art methods that considerably mitigate catastrophic forgetting do not necessarily decrease the stability gap well. One notable continual learning regularization approach, neuronal decay, attempts to encourage learning solutions that have small activations in the hidden layers. It previously showed improvement in terms of catastrophic forgetting but was not assessed in the context of stability gap. In this study, we compare neuronal decay with a baseline model to see if it can reduce the stability gap. Qualitative analysis with plots and quantitative analysis with metrics, such as gap depth, time-to-recover and average accuracy, both give strong evidence that this simple regularization method can reduce the stability gap with no substantial sacrifice of performance or training time.
The source code is available at https://github.com/zkkv/neuronal-decay.

Continual learning aims to enable neural networks to acquire new knowledge sequentially without forgetting what they have already learned. While many strategies have been developed to address catastrophic forgetting, a subtler challenge known as the stability gap—a temporary drop in performance immediately after switching tasks—remains insufficiently understood. Recent work suggests that the learning rate may influence this phenomenon by shaping the model’s optimization trajectory. This paper systematically investigates how different constant learning rates affect the stability gap and whether dynamic learning rate scheduling can mitigate it. Experiments on Rotated MNIST with perfect replay show that smaller constant learning rates reduce the immediate drop but slow down recovery and convergence, while larger rates yield higher final accuracy but at the cost of a more severe gap. Scheduling methods, including CyclicLR and our custom IncreaseLROnPlateau, demonstrate potential for balancing this trade-off, but also introduce new challenges such as intra-task fluctuations. Overall, a carefully tuned constant learning rate provides the most robust trade-off in this setting. By isolating and quantifying these effects, this work offers insights for selecting and tuning learning rates in continual learning and lays the groundwork for future studies on more effective scheduling strategies. All code and experiments are publicly available at: https://github.com/wjssk/learning-rate-in-stability-gap.

Support Vector Machines (SVMs) are widely used in various domains, with their performance heavily dependent on hyperparameter selection. However, hyperparameter tuning is computationally demanding due to the SVM training complexity, which is at best $O(n^2)$, where $n$ represents the number of training samples. To mitigate this challenge, we propose integrating a validation-based early stopping criterion into the Sequential Minimal Optimization (SMO) algorithm to enhance tuning efficiency.

We evaluate this approach within Random Search and Successive Halving frameworks, aiming to reduce tuning runtime while preserving model performance. We introduce a composite score function to facilitate a balanced assessment of accuracy and efficiency. Our empirical analysis reveals that incorporating early stopping into SMO significantly reduces hyperparameter tuning time under RS but provides limited benefits in successive halving, given its inherent efficiency. Additionally, while dataset characteristics influence the effectiveness of early stopping, we found evidence that dimensionality does not. We also observe that frequent early stopping objective assessments introduce computational overhead, which can offset runtime improvements. Reducing assessment frequency alleviates this issue, but it diminishes the effectiveness of early stopping.

Our findings highlight the potential of early stopping in SMO for optimizing SVM hyperparameter tuning, particularly within random search-based approaches. They also identify trade-offs in assessment frequency and dataset-specific factors.

Learning curves show the learning rate of a clas- sifier by plotting the dataset size used to train the classifier versus the error rate. By extrapolating these curves it is possible to predict how well the classifier will perform when trained on dataset sizes that are currently not available. This can be useful when trying to determine which classifier to select when dealing with a classification problem. Ob- taining these learning curves is usually done by fit- ting a parametric model to the learning data. This paper analyzes the potential of fitting the curve in a different space scaling the fitting data. This is done by analyzing the accuracy of the fit and the frequency of the fit succeeding. Our main findings are that log scaling produces better MSEs than not scaling, while exponential scaling is inconclusive.

Learning curves are graphical representations of the relationship between dataset size and error rate in machine learning. Curve fitting is the process of estimating a learning curve using a mathematical formula. This paper analyzes two ways of performing curve fitting: interpolation and extrapolation. The accuracy of the curve-fitting procedure might be negatively influenced by the irregular shape of the curve and the presence of noise. Our study investigates the effects of the Gaussian filter on curve fitting and the potential to improve its performance. This is done by analyzing multiple values of the Gaussian filter's standard deviation parameter(Sigma) and also a wide variety of learning curves(both smooth and noisy ones). The main finding of this research states that the Gaussian filter can generate significant improvements in the extrapolation process, especially when it is applied to noisy curves. On the other hand, for the interpolation procedure, its impact is reduced, even negligible for smooth curves. An important takeaway from this paper is that selecting the most suitable pre-processing method for the type of curve analyzed might generate valuable findings in the field of learning curves used in machine learning.

Learning curves are used to evaluate the perfor- mance of a machine learning (ML) model with respect to the amount of data used when train- ing. Curve fitting finds the unknown optimal co- efficients by minimizing the error prediction for a learning curve. This research analyzed the effect of parameter initialization on the performance of curve fitting. Our focus was on comparing the per- formance of sampling the initial parameters from 2 random distributions: uniform and normal on the curve fitting process for different parametric mod- els. Moreover, we looked into the effect of chang- ing the parameters for these 2 random distributions and drew conclusions about potential best initial guesses. Finally, we arrived at the conclusion that, after choosing parameters that maintain similar data dis- tribution, uniform and normal distribution sam- pling parameter initializations perform similarly during the curve-fitting process on learning curves. Moreover, our studies highlight the sensitivity of the Levenberg-Maquardt curve fitting method’s sensitivity to bad initial guesses.

The rapid advancement of artificial intelligence technologies has significantly increased the complexity of polymorphic and metamorphic malware, presenting new challenges to cybersecurity defenses. Our study introduces a novel bioinformatics-inspired approach, leveraging deep learning and phylogenetic analysis to understand the evolutionary dynamics of such malware. By analyzing a dataset of 103,883 malware samples, we transformed extracted features using pseudo-static, dynamic, and image analyses into embeddings with deep learning techniques, combining them into what we refer to as the "genome" of malware. These combined embeddings were used to construct phylogenetic trees employing the Unweighted Pair Group Method with Arithmetic Mean (UPGMA) and the Neighbor-Joining (NJ) method.We were the first to utilize OpenAI's state-of-the-art embeddings for converting pseudo-static and dynamic features into embeddings. In addition, we discovered that transfer learning with ResNet-50 is highly effective compared to traditional CNNs, producing better image embeddings that outperform others in terms of classification accuracy.

We also introduced new validation techniques for phylogenetic trees, making use of VirusTotal timestamps and embedding drift analysis. These methods confirmed that the NJ method was more accurate. Furthermore, we developed techniques to simplify the analysis of these extensive phylogenetic trees, enabling efficient derivation of relationships within and between malware families. The insights from our NJ-built phylogenetic trees closely align with public data and lay a foundation for generating evolutionary-informed signatures that enhance tailored detection strategies. Our method has significantly expedited the process of identifying connections among 538 malware families by dramatically reducing the timeframe from months or years to just weeks much faster than traditional reverse engineering approaches for tracing malware evolution.

Learning curves illustrate the relationship between the performance of learning algorithms and the increasing volume of training data [1, 2, 3]. While the concept of learning curves is well-established, clustering these curves based on fitting parameters remains an underexplored area. Our study delves into this domain and leverages the Learning Curve Database (LCDB) to discover potential patterns. We investigate whether different curve models uncover distinct patterns, examine the impact of different datasets on these learners, and explore if various learners display unique characteristics and behaviors or adhere to a common pattern. Curve model analyses conclude that most of the data points are in a single cluster (dominant cluster), indicating a potential commonality. Certain learners, such as QuadraticDiscriminantAnalysis and PassiveAggressiveClassifier, exhibit unique traits and do not conform to this common pattern, regardless of dataset attributes. Moreover, while various learners demonstrate similar characteristics within a single curve model, distinct patterns emerged when comparing across different curve models, indicating internal similarity but external divergence in behavior.

Learning curves are useful to determine the amount of data needed for a certain performance. The conventional belief is that increasing the amount of data improves performance. However, recent work challenges this assumption, and shows nonmonotonic behaviors of certain learners on certain problems. This paper presents a new approach for detecting non-monotonicity in empirical learning curves. This method monitors the degree of monotonicity violation on non-monotonic intervals, using the performance difference. In addition, the accuracy of the algorithm is being assessed through a series of diverse experiments. The proposed algorithm is applied to a subset of the extensive Learning Curve Database (LCDB). The results indicate an experimental accuracy of 95.5% in identifying non-monotonicity within real learning curves. Importantly, the metric demonstrated its ability to distinguish genuine non-monotonic trends from minor fluctuations attributed to measurement errors.

This study explores the extrapolation of learning curves, a crucial aspect in evaluating learner performance with varying dataset sample sizes. We use the Learning Curve Prior Fitted Network (LC-PFN), a transformer pre-trained on synthetic data with proficiency in approximate Bayesian inference, to investigate its predictive accuracy using the Learning Curve Database (LCDB). The assessment involves MSE as an error metric, with 2 baselines from previous studies where we see it outperform the baseline in some cases and keep on par in others. Additionally, we scrutinize instances where the LC-PFN model may exhibit shortcomings to identify trends in curve extrapolation failures, offering insights for potential modifications to the training dataset. We see a pattern in learners where LC-PFN performs consistently poorly on, whereas no significant pattern can be seen for datasets.

This research investigates the impact of data imbalances on the learning curve using the nearest mean model. Learning curves are useful to represent the performance of the model as the training size increases. Imbalanced datasets are often encountered in real-life scenarios and pose challenges to standard classifier models impacting their performance. Thus, the research question is ”How do data imbalances affect the learning curves using the nearest mean model?”. To answer the question, an experiment is conducted using data from a multivariate Gaussian distribution to sample varying levels of imbalances. The imbalance ratio explored is [0.1, 0.2, 0.3, 0.4, 0.5], representing the percentage of the dataset that consists of the minority class. The findings indicated that as the data becomes more imbalanced, the learning curves reach the accuracy plateau at a later rate. The analysis of the curve parameter which follows the logistic function suggests that imbalances have an impact on the maximum achievable accuracy and rightward shift of the curves. However, the maximum achievable accuracy is non-significant and the shape of the curves remains similar. Additionally, false negatives have a significant impact on the learning curves.