Microcontroller-based neural network inference faces significant RAM constraints, hindering performance and deployment. One of the main constraints is the peak memory usage, which is essential for conducting deep learning inferences with low latency. To address this, previous researchers have developed peak memory estimators to estimate the Peak Memory Usage, which could be used by inference-time optimization techniques like pruning to tackle the RAM constraint. But many of the peak memory estimators used by current state-of-the art frameworks like µNAS and TinyEngine produce underestimation or overestimation, reducing the reliability of model decisions made under RAM constraints. Underestimation often arises from failing to account for all components contributing to peak memory usage, while overestimation can occur when extra memory overheads irrelevant in MCU-specific inference scenarios are incorrectly included. In this paper, we propose our peak memory estimator, which estimates the peak memory usage of deep learning inference at the operator level and can accurately estimate the peak memory usage of a deep learning model during inference. The experiments show that our method achieves more accurate peak memory predictions across multiple MCU platforms and can be effectively integrated with pruning strategies to produce better model compression that satisfies both memory and accuracy constraints. Specifically, on a benchmark of 150 models, our method achieved an average estimation error margin of only 0.9%, significantly outperforming µNAS, which exhibited an average error margin of 61.7%.

There has been a steady increase in technologies that leverage Deep Learning (DL) techniques on resource-constrained devices for real-time processing. While DL techniques are adept at recognition tasks, their performance depends on the training process. Training data is seldom fully representative of the deployment scenario, requiring retraining to preserve accuracy. Many works have proposed on-device learning techniques that enable training DL techniques like Convolutional Neural Networks (CNNs) on microcontroller units (MCUs), without requiring data to be sent to the cloud. These methods have yielded promising accuracy improvements while training with low memory footprints. However, limited research has been done on the energy implications of doing so.

This thesis presents methods for energy-aware on-device learning on MCUs. It leverages the principle of updating specific layers of a CNN, proposed in past works, to fit memory constraints. We propose an energy-accuracy trade-off objective based on computational costs (in MACs) and accuracy improvement to select which layers to train. Furthermore, we demonstrate how computationally light search algorithms can adequately maximize the newly defined objective for layer selection. Evaluations show that our approach saves up to 200mJ of energy on-device while yielding simulation accuracies similar to a recent study under the same conditions.