T.J. Viering
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
52 records found
1
In this work, we extend nanoTabPFN, a Prior-Data Fitted Network (PFN) for tabular classification to produce Shapley attributions alongside its predictive distribution. The output is a bucketed distribution over signed feature attribution values. The class prediction is derived as the additive sum of a baseline and per-feature contributions, ensuring the explanation is faithful to the prediction by construction. The attribution head is supervised by TreeSHAP values from a decision tree prior, connecting the explanation to the data generating process (DGP). The resulting attributions are interpreted as a posterior over Shapley values. The PFN’s distributional mechanism is transferred to feature attributions to yield calibrated explanation uncertainty. Across synthetic and real tabular benchmarks, the model recovers attributions with high fidelity (R2 ≈ 0.94-0.95 against exact Shapley on synthetic data), but at a small cost to accuracy. It produces calibrated attribution uncertainty, with negative log-likelihood (NLL) below the uninformative uniform baseline.
...
In this work, we extend nanoTabPFN, a Prior-Data Fitted Network (PFN) for tabular classification to produce Shapley attributions alongside its predictive distribution. The output is a bucketed distribution over signed feature attribution values. The class prediction is derived as the additive sum of a baseline and per-feature contributions, ensuring the explanation is faithful to the prediction by construction. The attribution head is supervised by TreeSHAP values from a decision tree prior, connecting the explanation to the data generating process (DGP). The resulting attributions are interpreted as a posterior over Shapley values. The PFN’s distributional mechanism is transferred to feature attributions to yield calibrated explanation uncertainty. Across synthetic and real tabular benchmarks, the model recovers attributions with high fidelity (R2 ≈ 0.94-0.95 against exact Shapley on synthetic data), but at a small cost to accuracy. It produces calibrated attribution uncertainty, with negative log-likelihood (NLL) below the uninformative uniform baseline.
Learning to Curve
Exploring Metafeature-Driven Learning Curve Prediction
Learning curves in machine learning are useful tools to steer data acquisition and early stopping, representing the relation between training data quantity and performance. In existing literature, prediction of these learning curves largely relies on extrapolating from an initial set of observed performances and does not consider the characteristics of the underlying dataset, dubbed "metafeatures". Existing research of the utility of these metafeatures in the domain of learning curve prediction is sparse. Further, when it comes to measuring the fitness of predictions, the vast majority of literature uses Mean Squared Error. Different loss functions are seldom considered, and comparisons between different loss functions are not made.
In this Master Thesis we showcase metafeature-driven learning curve prediction by training two metalearners on a synthetic database of learning curves. We show that these metalearners strongly outperform the problem-average curve, which is indicative of meaningful task learning. The learning curves predictions produced by these metalearners perform comparable to curves obtained through direct curve fitting methods.
These results are obtained using 33 metafeatures. Four metafeature categories, comprising 12 metafeatures, are newly introduced in this research. Of these categories, we show high utility for three: Bayes Error, Negentropy and Surrogate Bayes. Of these, the Bayes Error shows highest utility. The best performing category of metafeature was the landmarkers, reinforcing their usage in literature.
We further show that the effect of chosen loss function on the shape of directly fitted curves on learning curve data is present but minor. The most notable difference appears in the exponent, where using a squared loss function yields a slightly lower (negative) exponent for power laws compared to the Mean Absolute Error. These trends do not readily appear to affect the shape of the resulting learning curve.
Evaluation or real-world data using the CC18-Database yielded no strong performance, suggesting that the synthetic learning curve database is not reflective of real problem datasets. Further research is needed to ascertain the nature of this discrepancy. ...
In this Master Thesis we showcase metafeature-driven learning curve prediction by training two metalearners on a synthetic database of learning curves. We show that these metalearners strongly outperform the problem-average curve, which is indicative of meaningful task learning. The learning curves predictions produced by these metalearners perform comparable to curves obtained through direct curve fitting methods.
These results are obtained using 33 metafeatures. Four metafeature categories, comprising 12 metafeatures, are newly introduced in this research. Of these categories, we show high utility for three: Bayes Error, Negentropy and Surrogate Bayes. Of these, the Bayes Error shows highest utility. The best performing category of metafeature was the landmarkers, reinforcing their usage in literature.
We further show that the effect of chosen loss function on the shape of directly fitted curves on learning curve data is present but minor. The most notable difference appears in the exponent, where using a squared loss function yields a slightly lower (negative) exponent for power laws compared to the Mean Absolute Error. These trends do not readily appear to affect the shape of the resulting learning curve.
Evaluation or real-world data using the CC18-Database yielded no strong performance, suggesting that the synthetic learning curve database is not reflective of real problem datasets. Further research is needed to ascertain the nature of this discrepancy. ...
Learning curves in machine learning are useful tools to steer data acquisition and early stopping, representing the relation between training data quantity and performance. In existing literature, prediction of these learning curves largely relies on extrapolating from an initial set of observed performances and does not consider the characteristics of the underlying dataset, dubbed "metafeatures". Existing research of the utility of these metafeatures in the domain of learning curve prediction is sparse. Further, when it comes to measuring the fitness of predictions, the vast majority of literature uses Mean Squared Error. Different loss functions are seldom considered, and comparisons between different loss functions are not made.
In this Master Thesis we showcase metafeature-driven learning curve prediction by training two metalearners on a synthetic database of learning curves. We show that these metalearners strongly outperform the problem-average curve, which is indicative of meaningful task learning. The learning curves predictions produced by these metalearners perform comparable to curves obtained through direct curve fitting methods.
These results are obtained using 33 metafeatures. Four metafeature categories, comprising 12 metafeatures, are newly introduced in this research. Of these categories, we show high utility for three: Bayes Error, Negentropy and Surrogate Bayes. Of these, the Bayes Error shows highest utility. The best performing category of metafeature was the landmarkers, reinforcing their usage in literature.
We further show that the effect of chosen loss function on the shape of directly fitted curves on learning curve data is present but minor. The most notable difference appears in the exponent, where using a squared loss function yields a slightly lower (negative) exponent for power laws compared to the Mean Absolute Error. These trends do not readily appear to affect the shape of the resulting learning curve.
Evaluation or real-world data using the CC18-Database yielded no strong performance, suggesting that the synthetic learning curve database is not reflective of real problem datasets. Further research is needed to ascertain the nature of this discrepancy.
In this Master Thesis we showcase metafeature-driven learning curve prediction by training two metalearners on a synthetic database of learning curves. We show that these metalearners strongly outperform the problem-average curve, which is indicative of meaningful task learning. The learning curves predictions produced by these metalearners perform comparable to curves obtained through direct curve fitting methods.
These results are obtained using 33 metafeatures. Four metafeature categories, comprising 12 metafeatures, are newly introduced in this research. Of these categories, we show high utility for three: Bayes Error, Negentropy and Surrogate Bayes. Of these, the Bayes Error shows highest utility. The best performing category of metafeature was the landmarkers, reinforcing their usage in literature.
We further show that the effect of chosen loss function on the shape of directly fitted curves on learning curve data is present but minor. The most notable difference appears in the exponent, where using a squared loss function yields a slightly lower (negative) exponent for power laws compared to the Mean Absolute Error. These trends do not readily appear to affect the shape of the resulting learning curve.
Evaluation or real-world data using the CC18-Database yielded no strong performance, suggesting that the synthetic learning curve database is not reflective of real problem datasets. Further research is needed to ascertain the nature of this discrepancy.
“Don’t tune hyperparameters on the test set” is often stated in machine learning textbooks. Violating it is considered a cardinal sin that produces misleadingly optimistic results, corrupts benchmark integrity, and thus can even be interpreted as scientific fraud. Yet evidence suggests that test set hyperparameter tuning does occur in practice, making it all the more important to understand its actual consequences. So how bad is it, really? In this work we question this dogma and put it to an empirical test. We systematically study the magnitude of the performance inflation caused by tuning the hyperparameters on the test set for MNIST-1D and CIFAR-10. Our experiments show that while the effect is real and significant, it is frequently small relative to other sources of noise. In many cases, we find that tuning on the test set recovers exactly the same model as when tuning on the validation set. Most importantly, we find that the rankings of models remain preserved after tuning on the test set, and therefore that consistent test-set tuning does not invalidate benchmarks or model selection. Our results call for a more nuanced view of tuning hyperparameters on the test set, stimulating researchers to openly report test tuning.
...
“Don’t tune hyperparameters on the test set” is often stated in machine learning textbooks. Violating it is considered a cardinal sin that produces misleadingly optimistic results, corrupts benchmark integrity, and thus can even be interpreted as scientific fraud. Yet evidence suggests that test set hyperparameter tuning does occur in practice, making it all the more important to understand its actual consequences. So how bad is it, really? In this work we question this dogma and put it to an empirical test. We systematically study the magnitude of the performance inflation caused by tuning the hyperparameters on the test set for MNIST-1D and CIFAR-10. Our experiments show that while the effect is real and significant, it is frequently small relative to other sources of noise. In many cases, we find that tuning on the test set recovers exactly the same model as when tuning on the validation set. Most importantly, we find that the rankings of models remain preserved after tuning on the test set, and therefore that consistent test-set tuning does not invalidate benchmarks or model selection. Our results call for a more nuanced view of tuning hyperparameters on the test set, stimulating researchers to openly report test tuning.
Classifying Packed Malware with Convolutional Neural Networks
A ScoreCAM and Occlusion Analysis of Necessary Features
Accurate and scalable detection has become a challenge in cybersecurity due to an exponential increase in the volume of new malware and its increasing complexity. A promising solution could be using convolutional neural networks to classify malware binaries interpreted as grayscale images. However, malware is frequently packed, which makes classification more difficult. We studied the impact of these packers on classification and what underlying features the models rely on. We trained independent ResNet-18 classifiers and evaluated them using two explainable AI methods: ScoreCAM and occlusion. The experimental setup analysed a synthetic dataset of 19,735 samples subjected to eleven different packers. We have four main results: (1) models perform poorly on large files packed with UPX, (2) models generalise well across structure-preserving packers, but (3) fail on transformations that alter the entire structural layout. Furthermore, (4) models rely more on unaffected binary sections when the main code section is obfuscated. Thus, convolutional neural networks remain highly dependent on file structure. Future research should investigate the efficacy of different explainable AI methods and the effects of resizing malware images.
...
Accurate and scalable detection has become a challenge in cybersecurity due to an exponential increase in the volume of new malware and its increasing complexity. A promising solution could be using convolutional neural networks to classify malware binaries interpreted as grayscale images. However, malware is frequently packed, which makes classification more difficult. We studied the impact of these packers on classification and what underlying features the models rely on. We trained independent ResNet-18 classifiers and evaluated them using two explainable AI methods: ScoreCAM and occlusion. The experimental setup analysed a synthetic dataset of 19,735 samples subjected to eleven different packers. We have four main results: (1) models perform poorly on large files packed with UPX, (2) models generalise well across structure-preserving packers, but (3) fail on transformations that alter the entire structural layout. Furthermore, (4) models rely more on unaffected binary sections when the main code section is obfuscated. Thus, convolutional neural networks remain highly dependent on file structure. Future research should investigate the efficacy of different explainable AI methods and the effects of resizing malware images.
Why Does Aggressive Resizing Preserve Malware Image Classification Performance?
Evaluating the Impact of Interpolation and Spatial Detail on Family-Discriminative Signals
Malware binaries can be represented as grayscale images by placing byte values on a two-dimensional grid. Convolutional neural networks can classify such malware images with high accuracy, but it is less clear why this performance can remain strong when the images are aggressively resized. This paper studies this phenomenon by examining how different interpolation methods affect accuracy, whether information learned by a high-resolution model remains useful after aggressive resizing, and which retained pixel values are most predictive for distinguishing malware families. The results show that all studied interpolation methods remain above 0.99 through 8 × 8. At lower resolutions, nearest-neighbour performs best at 4 × 4 and 2 × 2, whereas bilinear and bicubic unexpectedly perform better at 1 × 1. Blurring reduces accuracy, while a model trained at 224 × 224 does not transfer reliably when test images are downsampled and then restored to the original input size, indicating that broad texture alone is not sufficient for classification and that successful low-resolution models adapt to the resized representation. Decision-tree analysis further reveals that byte values sampled from fixed relative locations can contain family-specific signal, and backmapping shows that influential sampled locations frequently overlap with parsed binary sections, especially executable code. Still, these findings do not demonstrate semantic code understanding.
The results suggest that aggressive resizing preserves malware-family information through coarse byteplot layout, sampled byte values, and contrast patterns, rather than exact semantic binary regions. ...
The results suggest that aggressive resizing preserves malware-family information through coarse byteplot layout, sampled byte values, and contrast patterns, rather than exact semantic binary regions. ...
Malware binaries can be represented as grayscale images by placing byte values on a two-dimensional grid. Convolutional neural networks can classify such malware images with high accuracy, but it is less clear why this performance can remain strong when the images are aggressively resized. This paper studies this phenomenon by examining how different interpolation methods affect accuracy, whether information learned by a high-resolution model remains useful after aggressive resizing, and which retained pixel values are most predictive for distinguishing malware families. The results show that all studied interpolation methods remain above 0.99 through 8 × 8. At lower resolutions, nearest-neighbour performs best at 4 × 4 and 2 × 2, whereas bilinear and bicubic unexpectedly perform better at 1 × 1. Blurring reduces accuracy, while a model trained at 224 × 224 does not transfer reliably when test images are downsampled and then restored to the original input size, indicating that broad texture alone is not sufficient for classification and that successful low-resolution models adapt to the resized representation. Decision-tree analysis further reveals that byte values sampled from fixed relative locations can contain family-specific signal, and backmapping shows that influential sampled locations frequently overlap with parsed binary sections, especially executable code. Still, these findings do not demonstrate semantic code understanding.
The results suggest that aggressive resizing preserves malware-family information through coarse byteplot layout, sampled byte values, and contrast patterns, rather than exact semantic binary regions.
The results suggest that aggressive resizing preserves malware-family information through coarse byteplot layout, sampled byte values, and contrast patterns, rather than exact semantic binary regions.
Bayesian optimization for hyperparameter optimization (HPO) and combined algorithm selection and hyperparameter optimization (CASH) often relies on explicit hyperparameter encodings as surrogate inputs. In high-dimensional search spaces, however, such representations can make it difficult for standard surrogates to learn useful performance models. We instead study an alternative metalearning representation that describes each configuration directly as its historical performance across prior tasks and can be paired with standard surrogate models. We further apply this representation to maximal marginal relevance (MMR), a diversity-based re-ranking criterion from recommender systems, as a posterior-free acquisition rule. When selecting and optimising across multiple models on the TabArena benchmark, the performance-based representation paired with a linear Bayesian ridge regression model outperforms dedicated meta-learning baselines. On single-model HPO it remains a competitive second. Overall this method closely matches or improves on regret at a fraction of the complexity and cost, requiring no pre-training or tuning and scaling independently of search space dimensionality.
...
Bayesian optimization for hyperparameter optimization (HPO) and combined algorithm selection and hyperparameter optimization (CASH) often relies on explicit hyperparameter encodings as surrogate inputs. In high-dimensional search spaces, however, such representations can make it difficult for standard surrogates to learn useful performance models. We instead study an alternative metalearning representation that describes each configuration directly as its historical performance across prior tasks and can be paired with standard surrogate models. We further apply this representation to maximal marginal relevance (MMR), a diversity-based re-ranking criterion from recommender systems, as a posterior-free acquisition rule. When selecting and optimising across multiple models on the TabArena benchmark, the performance-based representation paired with a linear Bayesian ridge regression model outperforms dedicated meta-learning baselines. On single-model HPO it remains a competitive second. Overall this method closely matches or improves on regret at a fraction of the complexity and cost, requiring no pre-training or tuning and scaling independently of search space dimensionality.
CNN-based malware byteplot classifiers achieve high accuracy under clean conditions, but their robustness to input perturbations remains poorly understood. This paper systematically evaluates how a ResNet18 classifier degrades under standard image transformations (rotations, brightness and contrast shifts, and flips) applied to both grayscale and RGB byteplot representations of a 14-class, 20,020 image dataset. Each transformation is grounded in a realistic attacker model via its correspondence to a binary level obfuscation technique. Results show that rotations are catastrophic even at small angles, flips exhibit strong asymmetry driven by the vertical structure of byteplot sections, and photometric shifts are tolerated until a certain threshold. Training on a mixed set that includes 25% distorted images substantially recovers robustness across all categories, with gains being largely self-attributing to their matching transformation type. RGB representations amplify whatever performance trend is already present in grayscale, for better or worse. Together, the findings reveal that CNN byteplot classifiers exploit specific spatial and photometric properties of their input, and that targeted augmented training can harden them against realistic evasion strategies without sacrificing clean image performance.
...
CNN-based malware byteplot classifiers achieve high accuracy under clean conditions, but their robustness to input perturbations remains poorly understood. This paper systematically evaluates how a ResNet18 classifier degrades under standard image transformations (rotations, brightness and contrast shifts, and flips) applied to both grayscale and RGB byteplot representations of a 14-class, 20,020 image dataset. Each transformation is grounded in a realistic attacker model via its correspondence to a binary level obfuscation technique. Results show that rotations are catastrophic even at small angles, flips exhibit strong asymmetry driven by the vertical structure of byteplot sections, and photometric shifts are tolerated until a certain threshold. Training on a mixed set that includes 25% distorted images substantially recovers robustness across all categories, with gains being largely self-attributing to their matching transformation type. RGB representations amplify whatever performance trend is already present in grayscale, for better or worse. Together, the findings reveal that CNN byteplot classifiers exploit specific spatial and photometric properties of their input, and that targeted augmented training can harden them against realistic evasion strategies without sacrificing clean image performance.
Malware image classification using CNNs has achieved classification accuracies of up to 99%, but the black-box nature of these models limits their trustworthiness. Explainable AI (xAI) methods can highlight which image regions drive a model's predictions, yet prior work has not mapped these regions back to named binary sections or quantitatively validated their importance. This paper addresses both gaps by applying Grad-CAM, LIME, and SHAP to a ResNet-18 classifier. For each test sample, highlighted image regions are mapped to named binary sections, and an occlusion experiment validates that high-attention sections are genuinely critical to classification. All three methods consistently rank .rdata, .data, and .eh_frame as the most important sections for PE binary classification, with attention distributed more broadly across ELF binaries. Occluding the top three sections per family causes accuracy to drop below 50% across all methods, confirming the importance of the identified regions. SHAP identifies the single most important section most precisely, while Grad-CAM achieves comparable overall performance at 50–60 times the speed. Notably, high classifier attention on .eh_frame suggests that the model may exploit tool-specific artifacts, raising concerns that should be accounted for before deploying these classifiers in production.
...
Malware image classification using CNNs has achieved classification accuracies of up to 99%, but the black-box nature of these models limits their trustworthiness. Explainable AI (xAI) methods can highlight which image regions drive a model's predictions, yet prior work has not mapped these regions back to named binary sections or quantitatively validated their importance. This paper addresses both gaps by applying Grad-CAM, LIME, and SHAP to a ResNet-18 classifier. For each test sample, highlighted image regions are mapped to named binary sections, and an occlusion experiment validates that high-attention sections are genuinely critical to classification. All three methods consistently rank .rdata, .data, and .eh_frame as the most important sections for PE binary classification, with attention distributed more broadly across ELF binaries. Occluding the top three sections per family causes accuracy to drop below 50% across all methods, confirming the importance of the identified regions. SHAP identifies the single most important section most precisely, while Grad-CAM achieves comparable overall performance at 50–60 times the speed. Notably, high classifier attention on .eh_frame suggests that the model may exploit tool-specific artifacts, raising concerns that should be accounted for before deploying these classifiers in production.
Prior specification is a fundamental challenge in Bayesian inference. Traditionally, a prior represents a belief over the behavior of a statistical model based on expert knowledge. Such knowledge is not always available, or can be hard to express in a numerical form. As an answer, this thesis introduces the Prior-Learning Prior-Fitted Network (PLPFN), a transformer-based meta-learning model that amortizes prior learning. The model takes in a set of related tasks and outputs fully Bayesian uncertainty estimates over prior parameters in a single forward pass. The novelty lies in providing interpretable prior parameter outputs and examining the prior learning problem through the lens of hierarchical modeling and identifiability. The framework is evaluated on two representative prior structures: Bayesian Linear Regression and a Hierarchical Gaussian Process. The results show that PLPFN's approximations remain close to asymptotically exact baselines, such as closed-form conjugate prior learning and Markov Chain Monte Carlo. Applying the learned priors to Bayesian Optimization shows that the PLPFN is competitive with state-of-the-art prior learning methods in terms of sample efficiency and final regret, both in- and out-of-distribution. The PLPFN framework is a valuable step in shifting prior specification in Bayesian inference from the complex manual translation of beliefs to specifying a set of related datasets.
...
Prior specification is a fundamental challenge in Bayesian inference. Traditionally, a prior represents a belief over the behavior of a statistical model based on expert knowledge. Such knowledge is not always available, or can be hard to express in a numerical form. As an answer, this thesis introduces the Prior-Learning Prior-Fitted Network (PLPFN), a transformer-based meta-learning model that amortizes prior learning. The model takes in a set of related tasks and outputs fully Bayesian uncertainty estimates over prior parameters in a single forward pass. The novelty lies in providing interpretable prior parameter outputs and examining the prior learning problem through the lens of hierarchical modeling and identifiability. The framework is evaluated on two representative prior structures: Bayesian Linear Regression and a Hierarchical Gaussian Process. The results show that PLPFN's approximations remain close to asymptotically exact baselines, such as closed-form conjugate prior learning and Markov Chain Monte Carlo. Applying the learned priors to Bayesian Optimization shows that the PLPFN is competitive with state-of-the-art prior learning methods in terms of sample efficiency and final regret, both in- and out-of-distribution. The PLPFN framework is a valuable step in shifting prior specification in Bayesian inference from the complex manual translation of beliefs to specifying a set of related datasets.
Teaching Machines to Critique Computer Science Theses
A Human-in-the-Loop Framework for Discipline-Aware, Span-Anchored LLM Feedback in Thesis Supervision
To produce reviewable draft feedback on long computer-science thesis drafts under supervisor control, we present a controlled three-way comparison of LLM generation strategies on open-weight backbones, paired with a supervisor-facing PDF review interface. We compare base and LoRA-fine-tuned Llama 3.3 70B and Qwen 3.5 27B across whole-document, section-aware two-stage, and agentic review strategies, evaluated through an LLM-as-judge benchmark over all twelve configurations, a blind human rating study, and an interface usability study. Whole-document generation obtains the highest judge-macro score. Fine-tuning helps in only one of six matched base-versus-fine-tuned comparisons, indicating that supervised adaptation on a small span-anchored corpus shifts response distribution without expanding critique ability. In the blind study, the deployed section-aware generation is rated significantly higher than held-out supervisor feedback on change clarity (p = .027) and specificity (p = .018), while correctness shows no significant difference (p = .516). On the comments scored by both, three independent LLM judges rate the system above the human raters on every dimension (mean bias 0.41 points), most strongly on supervisor suitability. The PDF review interface reaches a mean System Usability Scale (SUS) score of 77.5. The results support using open-weight LLMs to produce reviewable draft feedback on long technical theses under supervisor inspection, editing, and export control.
...
To produce reviewable draft feedback on long computer-science thesis drafts under supervisor control, we present a controlled three-way comparison of LLM generation strategies on open-weight backbones, paired with a supervisor-facing PDF review interface. We compare base and LoRA-fine-tuned Llama 3.3 70B and Qwen 3.5 27B across whole-document, section-aware two-stage, and agentic review strategies, evaluated through an LLM-as-judge benchmark over all twelve configurations, a blind human rating study, and an interface usability study. Whole-document generation obtains the highest judge-macro score. Fine-tuning helps in only one of six matched base-versus-fine-tuned comparisons, indicating that supervised adaptation on a small span-anchored corpus shifts response distribution without expanding critique ability. In the blind study, the deployed section-aware generation is rated significantly higher than held-out supervisor feedback on change clarity (p = .027) and specificity (p = .018), while correctness shows no significant difference (p = .516). On the comments scored by both, three independent LLM judges rate the system above the human raters on every dimension (mean bias 0.41 points), most strongly on supervisor suitability. The PDF review interface reaches a mean System Usability Scale (SUS) score of 77.5. The results support using open-weight LLMs to produce reviewable draft feedback on long technical theses under supervisor inspection, editing, and export control.
Malware Through the Lens of Computer Vision
How Binary-to-Image Encodings Influence CNN-Based Malware Family Classification
Image-based malware classification through binary-to-image encodings has become a popular, quick-to-run and disassembly-free alternative to signature-based methods. Near-perfect accuracies are often reported, alongside weakly substantiated claims of resilience to malware obfuscation by adversaries. We present the first controlled, head-to-head comparison of four such encodings - grayscale and RGB byteplots, Markov bigram plots, and sliding-window Shannon entropy - under a fixed ResNet-18 classifier and a balanced dataset spanning seven packer conditions. To test whether high accuracy reflects genuine family structure or exploitable shortcuts, we pair the comparison with a diagnostic framework combining random- forest feature baselines, TLSH similarity analysis, HiResCAM and occlusion-based explainability methods. No single diagnostic suffices; only their combination separates shortcut-driven scores from genuine learning. We find that encoding performance cannot be judged by accuracy alone; models may fit to shortcuts such as file-size and near-duplicate samples, rather than learning malware structure. Entropy is found to be the dominant and most obfuscation-resilient encoding, while exhibiting the most evidence of genuine family-discriminative learning.
...
Image-based malware classification through binary-to-image encodings has become a popular, quick-to-run and disassembly-free alternative to signature-based methods. Near-perfect accuracies are often reported, alongside weakly substantiated claims of resilience to malware obfuscation by adversaries. We present the first controlled, head-to-head comparison of four such encodings - grayscale and RGB byteplots, Markov bigram plots, and sliding-window Shannon entropy - under a fixed ResNet-18 classifier and a balanced dataset spanning seven packer conditions. To test whether high accuracy reflects genuine family structure or exploitable shortcuts, we pair the comparison with a diagnostic framework combining random- forest feature baselines, TLSH similarity analysis, HiResCAM and occlusion-based explainability methods. No single diagnostic suffices; only their combination separates shortcut-driven scores from genuine learning. We find that encoding performance cannot be judged by accuracy alone; models may fit to shortcuts such as file-size and near-duplicate samples, rather than learning malware structure. Entropy is found to be the dominant and most obfuscation-resilient encoding, while exhibiting the most evidence of genuine family-discriminative learning.
Motivation: Clustering is an unsupervised learning task with broad applications. Traditional clustering methods often rely on point estimates of model parameters, which can limit their ability to capture uncertainty. Bayesian clustering addresses this by incorporating uncertainty into parameter estimation. However, existing Bayesian inference methods like Markov Chain Monte Carlo and Variational Inference are computationally intensive and can produce biased approximations. To overcome these limitations, we propose Cluster-PFN, a transformer-based model inspired by Prior-Data Fitted Networks. Cluster-PFN simultaneously approximates the posterior distributions over the cluster assignments of individual data points and the total number of clusters for the given dataset in a single forward pass. Our model provides fast and accurate Bayesian clustering, supporting data with up to five features and conditioning on user-specified cluster counts.
Results: Our results demonstrate that Cluster-PFN can predict the number of clusters up to 20% more accurately than standard heuristics. It also outperforms the Bayesian Gaussian Mixture Model using Variational Inference (Bayesian GMM VI), achieving up to 60% higher scores on certain external metrics while being up to 20 times faster during inference. Additionally, Cluster-PFN surpasses both the traditional Gaussian Mixture Model and K-means++ across the same external evaluation metrics. ...
Results: Our results demonstrate that Cluster-PFN can predict the number of clusters up to 20% more accurately than standard heuristics. It also outperforms the Bayesian Gaussian Mixture Model using Variational Inference (Bayesian GMM VI), achieving up to 60% higher scores on certain external metrics while being up to 20 times faster during inference. Additionally, Cluster-PFN surpasses both the traditional Gaussian Mixture Model and K-means++ across the same external evaluation metrics. ...
Motivation: Clustering is an unsupervised learning task with broad applications. Traditional clustering methods often rely on point estimates of model parameters, which can limit their ability to capture uncertainty. Bayesian clustering addresses this by incorporating uncertainty into parameter estimation. However, existing Bayesian inference methods like Markov Chain Monte Carlo and Variational Inference are computationally intensive and can produce biased approximations. To overcome these limitations, we propose Cluster-PFN, a transformer-based model inspired by Prior-Data Fitted Networks. Cluster-PFN simultaneously approximates the posterior distributions over the cluster assignments of individual data points and the total number of clusters for the given dataset in a single forward pass. Our model provides fast and accurate Bayesian clustering, supporting data with up to five features and conditioning on user-specified cluster counts.
Results: Our results demonstrate that Cluster-PFN can predict the number of clusters up to 20% more accurately than standard heuristics. It also outperforms the Bayesian Gaussian Mixture Model using Variational Inference (Bayesian GMM VI), achieving up to 60% higher scores on certain external metrics while being up to 20 times faster during inference. Additionally, Cluster-PFN surpasses both the traditional Gaussian Mixture Model and K-means++ across the same external evaluation metrics.
Results: Our results demonstrate that Cluster-PFN can predict the number of clusters up to 20% more accurately than standard heuristics. It also outperforms the Bayesian Gaussian Mixture Model using Variational Inference (Bayesian GMM VI), achieving up to 60% higher scores on certain external metrics while being up to 20 times faster during inference. Additionally, Cluster-PFN surpasses both the traditional Gaussian Mixture Model and K-means++ across the same external evaluation metrics.
One of the problems in continual learning, where models are trained sequentially on tasks, is a sudden drop in performance after switching to a new task, called stability gap. The presence of stability gap likely indicates that training is not done optimally. In this work we aim to address stability gap problem by using sharpness-aware optimization that biases convergence to flat minima. While flat minima are known to mitigate forgetting, their role in ensuring stable learning during task transitions remains unexplored. Through systematic analysis of two Entropy-SGD and C-Flat, we demonstrate that sharpness-aware optimizers produce smoother learning trajectories with reduced instability after task switch. Furthermore, we show that C-Flat’s second-order curvature approximation provides additional stabilization, suggesting that efficient Hessian-aware methods offer advantages for continual learning. The source code is available at Stability-Gap-SAM.
...
One of the problems in continual learning, where models are trained sequentially on tasks, is a sudden drop in performance after switching to a new task, called stability gap. The presence of stability gap likely indicates that training is not done optimally. In this work we aim to address stability gap problem by using sharpness-aware optimization that biases convergence to flat minima. While flat minima are known to mitigate forgetting, their role in ensuring stable learning during task transitions remains unexplored. Through systematic analysis of two Entropy-SGD and C-Flat, we demonstrate that sharpness-aware optimizers produce smoother learning trajectories with reduced instability after task switch. Furthermore, we show that C-Flat’s second-order curvature approximation provides additional stabilization, suggesting that efficient Hessian-aware methods offer advantages for continual learning. The source code is available at Stability-Gap-SAM.
Learning curve extrapolation helps practitioners predict model performance at larger data scales, enabling better planning for data collection and computational resource allocation. This paper investigates when neural networks outperform parametric models for this task. We conduct a comprehensive comparison of LC-PFNs (Learning Curve Prior-Fitted Networks) and three established parametric models (POW4, MMF4, WBL4) using LCDB v1.1, a large-scale dataset with learning curves generated across 265 classification tasks and 24 learners. Surprisingly, we find that parametric models — especially POW4 and MMF4 — consistently outperform LC-PFN across all generalization scenarios and most cutoff regions. However, LC-PFN demonstrates competitive performance when extrapolating from early-stage data, ranking second-best at 10%, 30%, and 50% cutoffs. This suggests LC-PFNs can be valuable when only a small fraction of the learning curve is available. LC-PFN is particularly challenged by smooth and flat curves, but shows slightly improved performance on irregular patterns such as peaking and dipping curves, though it remains outperformed by all parametric models. These trends highlight a misalignment between LC-PFN’s training distribution and the real-world diversity of learning curves. Our findings emphasize the strength of parametric models under realistic conditions and suggest avenues for improving LC-PFNs through architectural flexibility and curve length variability during training.
...
Learning curve extrapolation helps practitioners predict model performance at larger data scales, enabling better planning for data collection and computational resource allocation. This paper investigates when neural networks outperform parametric models for this task. We conduct a comprehensive comparison of LC-PFNs (Learning Curve Prior-Fitted Networks) and three established parametric models (POW4, MMF4, WBL4) using LCDB v1.1, a large-scale dataset with learning curves generated across 265 classification tasks and 24 learners. Surprisingly, we find that parametric models — especially POW4 and MMF4 — consistently outperform LC-PFN across all generalization scenarios and most cutoff regions. However, LC-PFN demonstrates competitive performance when extrapolating from early-stage data, ranking second-best at 10%, 30%, and 50% cutoffs. This suggests LC-PFNs can be valuable when only a small fraction of the learning curve is available. LC-PFN is particularly challenged by smooth and flat curves, but shows slightly improved performance on irregular patterns such as peaking and dipping curves, though it remains outperformed by all parametric models. These trends highlight a misalignment between LC-PFN’s training distribution and the real-world diversity of learning curves. Our findings emphasize the strength of parametric models under realistic conditions and suggest avenues for improving LC-PFNs through architectural flexibility and curve length variability during training.
How Noisy Is Too Noisy?
Robust Extrapolation of Learning Curves with LC-PFN
Accurately predicting a machine learning model’s final performance based on only partial training data can save substantial computational resources and guide early stopping, model selection, and automated machine learning (AutoML) workflows. Learning Curve Prior-Fitted Networks (LC-PFNs) are a recent data-driven approach to this problem, leveraging transformers trained on prior learning curves to make extrapolations. However, real-world training logs are often noisy and irregular—conditions under which the reliability of LC-PFNs remains largely untested. This thesis presents a systematic investigation into the robustness of LC-PFNs when exposed to noisy input data. Using LCDB 1.1—a large-scale dataset—we simulate various levels of noise by corrupting the input data with Gaussian perturbations and quantify how prediction accuracy degrades. To improve resilience, we study and propose two complementary mitigation strategies. The first injects artificial noise into the training data itself, either at a constant level or increasing gradually throughout training, encouraging the model to generalize across a spectrum of noise conditions. The second applies post-processing techniques at inference time, such as smoothing the input sequence with an exponential moving average or averaging multiple stochastic predictions using dropout. Our results show that standard LC-PFNs, trained only on clean data, are highly sensitive to even minor corruptions. In contrast, models trained with gradual noise exposure and evaluated with input smoothing achieve much greater robustness—reducing error by up to 75% under severe noise—while maintaining high accuracy in noise-free settings. This work demonstrates that substantial improvements in learning-curve extrapolation can be achieved without modifying model architecture, using general purpose techniques suitable for real-world deployment.
...
Accurately predicting a machine learning model’s final performance based on only partial training data can save substantial computational resources and guide early stopping, model selection, and automated machine learning (AutoML) workflows. Learning Curve Prior-Fitted Networks (LC-PFNs) are a recent data-driven approach to this problem, leveraging transformers trained on prior learning curves to make extrapolations. However, real-world training logs are often noisy and irregular—conditions under which the reliability of LC-PFNs remains largely untested. This thesis presents a systematic investigation into the robustness of LC-PFNs when exposed to noisy input data. Using LCDB 1.1—a large-scale dataset—we simulate various levels of noise by corrupting the input data with Gaussian perturbations and quantify how prediction accuracy degrades. To improve resilience, we study and propose two complementary mitigation strategies. The first injects artificial noise into the training data itself, either at a constant level or increasing gradually throughout training, encouraging the model to generalize across a spectrum of noise conditions. The second applies post-processing techniques at inference time, such as smoothing the input sequence with an exponential moving average or averaging multiple stochastic predictions using dropout. Our results show that standard LC-PFNs, trained only on clean data, are highly sensitive to even minor corruptions. In contrast, models trained with gradual noise exposure and evaluated with input smoothing achieve much greater robustness—reducing error by up to 75% under severe noise—while maintaining high accuracy in noise-free settings. This work demonstrates that substantial improvements in learning-curve extrapolation can be achieved without modifying model architecture, using general purpose techniques suitable for real-world deployment.
Learning curves represent the relationship between the amount of training data and the error rate in machine learning. An important use case for learning curves is extrapolating them in order to predict how much data is needed to achieve a certain performance. One way to do such extrapolations is using Deep Learning with a Prior-Fitted Network(PFN). This paper explores how training the PFN on an imbalanced dataset, i.e. containing learning curves from two or more machine learning models with a skewed distribution, affects the performance of the network. Research into imbalanced learning has shown that machine learning models can favor the more prevalent classes or data. Therefore, it is worthwhile to explore whether such trends can occur for the neural networks that we train for learning curve extrapolation. Our experiments focused on analyzing different imbalance scenarios and comparing them. Our results show that mixing learning curves from different learners can improve extrapolation performance in some cases, but the effect strongly depends on the learner characteristics and training proportions.
...
Learning curves represent the relationship between the amount of training data and the error rate in machine learning. An important use case for learning curves is extrapolating them in order to predict how much data is needed to achieve a certain performance. One way to do such extrapolations is using Deep Learning with a Prior-Fitted Network(PFN). This paper explores how training the PFN on an imbalanced dataset, i.e. containing learning curves from two or more machine learning models with a skewed distribution, affects the performance of the network. Research into imbalanced learning has shown that machine learning models can favor the more prevalent classes or data. Therefore, it is worthwhile to explore whether such trends can occur for the neural networks that we train for learning curve extrapolation. Our experiments focused on analyzing different imbalance scenarios and comparing them. Our results show that mixing learning curves from different learners can improve extrapolation performance in some cases, but the effect strongly depends on the learner characteristics and training proportions.
I Fought the Low
Decreasing Stability Gap with Neuronal Decay
Task-based continual learning setups suffer from temporary dips in performance shortly after switching to new tasks, a phenomenon referred to as stability gap. State-of-the-art methods that considerably mitigate catastrophic forgetting do not necessarily decrease the stability gap well. One notable continual learning regularization approach, neuronal decay, attempts to encourage learning solutions that have small activations in the hidden layers. It previously showed improvement in terms of catastrophic forgetting but was not assessed in the context of stability gap. In this study, we compare neuronal decay with a baseline model to see if it can reduce the stability gap. Qualitative analysis with plots and quantitative analysis with metrics, such as gap depth, time-to-recover and average accuracy, both give strong evidence that this simple regularization method can reduce the stability gap with no substantial sacrifice of performance or training time.
The source code is available at https://github.com/zkkv/neuronal-decay. ...
The source code is available at https://github.com/zkkv/neuronal-decay. ...
Task-based continual learning setups suffer from temporary dips in performance shortly after switching to new tasks, a phenomenon referred to as stability gap. State-of-the-art methods that considerably mitigate catastrophic forgetting do not necessarily decrease the stability gap well. One notable continual learning regularization approach, neuronal decay, attempts to encourage learning solutions that have small activations in the hidden layers. It previously showed improvement in terms of catastrophic forgetting but was not assessed in the context of stability gap. In this study, we compare neuronal decay with a baseline model to see if it can reduce the stability gap. Qualitative analysis with plots and quantitative analysis with metrics, such as gap depth, time-to-recover and average accuracy, both give strong evidence that this simple regularization method can reduce the stability gap with no substantial sacrifice of performance or training time.
The source code is available at https://github.com/zkkv/neuronal-decay.
The source code is available at https://github.com/zkkv/neuronal-decay.
Continual learning aims to enable neural networks to acquire new knowledge sequentially without forgetting what they have already learned. While many strategies have been developed to address catastrophic forgetting, a subtler challenge known as the stability gap—a temporary drop in performance immediately after switching tasks—remains insufficiently understood. Recent work suggests that the learning rate may influence this phenomenon by shaping the model’s optimization trajectory. This paper systematically investigates how different constant learning rates affect the stability gap and whether dynamic learning rate scheduling can mitigate it. Experiments on Rotated MNIST with perfect replay show that smaller constant learning rates reduce the immediate drop but slow down recovery and convergence, while larger rates yield higher final accuracy but at the cost of a more severe gap. Scheduling methods, including CyclicLR and our custom IncreaseLROnPlateau, demonstrate potential for balancing this trade-off, but also introduce new challenges such as intra-task fluctuations. Overall, a carefully tuned constant learning rate provides the most robust trade-off in this setting. By isolating and quantifying these effects, this work offers insights for selecting and tuning learning rates in continual learning and lays the groundwork for future studies on more effective scheduling strategies. All code and experiments are publicly available at: https://github.com/wjssk/learning-rate-in-stability-gap.
...
Continual learning aims to enable neural networks to acquire new knowledge sequentially without forgetting what they have already learned. While many strategies have been developed to address catastrophic forgetting, a subtler challenge known as the stability gap—a temporary drop in performance immediately after switching tasks—remains insufficiently understood. Recent work suggests that the learning rate may influence this phenomenon by shaping the model’s optimization trajectory. This paper systematically investigates how different constant learning rates affect the stability gap and whether dynamic learning rate scheduling can mitigate it. Experiments on Rotated MNIST with perfect replay show that smaller constant learning rates reduce the immediate drop but slow down recovery and convergence, while larger rates yield higher final accuracy but at the cost of a more severe gap. Scheduling methods, including CyclicLR and our custom IncreaseLROnPlateau, demonstrate potential for balancing this trade-off, but also introduce new challenges such as intra-task fluctuations. Overall, a carefully tuned constant learning rate provides the most robust trade-off in this setting. By isolating and quantifying these effects, this work offers insights for selecting and tuning learning rates in continual learning and lays the groundwork for future studies on more effective scheduling strategies. All code and experiments are publicly available at: https://github.com/wjssk/learning-rate-in-stability-gap.
In the context of continual learning, recent work has identified a significant and recurring perfor- mance drop, followed by a gradual recovery, upon the introduction of a new task. This phenomenon is referred to as the stability gap. Investigating it and the potential solutions is essential, as such findings can reduce both the energy consumption and computational time required to prepare a high- performing agent. Given the strong influence of training procedures on model performance and sta- bility, we analyze how various optimizers –SGD, NAG, AdaGrad, RMSprop, Adam– and momentum values affect the stability gap. We expose a deep neural network to a sequence of digit-identification tasks with varying rotations, and track several met- rics to capture the components of the stability gap and the overall performance. Our results reveal that increasing momentum amplifies the steepness and depth of the gap, while shortening its duration. Within this simplified setup, RMSprop proves most effective in reducing the magnitude and duration of the drop while maintaining high overall perfor- mance.
...
In the context of continual learning, recent work has identified a significant and recurring perfor- mance drop, followed by a gradual recovery, upon the introduction of a new task. This phenomenon is referred to as the stability gap. Investigating it and the potential solutions is essential, as such findings can reduce both the energy consumption and computational time required to prepare a high- performing agent. Given the strong influence of training procedures on model performance and sta- bility, we analyze how various optimizers –SGD, NAG, AdaGrad, RMSprop, Adam– and momentum values affect the stability gap. We expose a deep neural network to a sequence of digit-identification tasks with varying rotations, and track several met- rics to capture the components of the stability gap and the overall performance. Our results reveal that increasing momentum amplifies the steepness and depth of the gap, while shortening its duration. Within this simplified setup, RMSprop proves most effective in reducing the magnitude and duration of the drop while maintaining high overall perfor- mance.
Effectiveness of Machine Learning Models in Classifying Learners Based on Learning Curves
Improving Our Understanding of Learning Curves Through the Process of Classification
In machine learning, learning curves are a metric that plots performance versus training set size. They inform decisions about data acquisition, model selection, and hyperparameter tuning. Despite their importance, recent research suggests that our understanding of learning curve behavior remains limited. In this work, we investigate learning curves from a classification perspective to better understand their structural properties. By framing learning curves as time series and applying time series classification (TSC) techniques, we uncover several key findings: (1) training accuracy curves are significantly more distinguishable across models than validation or test curves; (2) learning curves become more informative and discriminative after a sufficient number of anchor points; and (3) TSC models that emphasize global structural features outperform those focused on local or pointwise characteristics. These results not only offer new insights into the nature of learning curves but also suggest promising directions for future work, including the development of specialized models that move beyond conventional time series assumptions.
...
In machine learning, learning curves are a metric that plots performance versus training set size. They inform decisions about data acquisition, model selection, and hyperparameter tuning. Despite their importance, recent research suggests that our understanding of learning curve behavior remains limited. In this work, we investigate learning curves from a classification perspective to better understand their structural properties. By framing learning curves as time series and applying time series classification (TSC) techniques, we uncover several key findings: (1) training accuracy curves are significantly more distinguishable across models than validation or test curves; (2) learning curves become more informative and discriminative after a sufficient number of anchor points; and (3) TSC models that emphasize global structural features outperform those focused on local or pointwise characteristics. These results not only offer new insights into the nature of learning curves but also suggest promising directions for future work, including the development of specialized models that move beyond conventional time series assumptions.