Neural network relief: a pruning algorithm based on neural activity

Journal Article (2024)
Author(s)

A. Dekhovich (TU Delft - Team Marcel Sluiter)

David M.J. Tax (TU Delft - Pattern Recognition and Bioinformatics)

M. H.F. Sluiter (TU Delft - Team Marcel Sluiter)

M. A. Bessa (Brown University)

Research Group
Team Marcel Sluiter
Copyright
© 2024 A. Dekhovich, D.M.J. Tax, M.H.F. Sluiter, M.A. Bessa
DOI related publication
https://doi.org/10.1007/s10994-024-06516-z
More Info
expand_more
Publication Year
2024
Language
English
Copyright
© 2024 A. Dekhovich, D.M.J. Tax, M.H.F. Sluiter, M.A. Bessa
Research Group
Team Marcel Sluiter
Issue number
5
Volume number
113
Pages (from-to)
2597-2618
Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Abstract

Current deep neural networks (DNNs) are overparameterized and use most of their neuronal connections during inference for each task. The human brain, however, developed specialized regions for different tasks and performs inference with a small fraction of its neuronal connections. We propose an iterative pruning strategy introducing a simple importance-score metric that deactivates unimportant connections, tackling overparameterization in DNNs and modulating the firing patterns. The aim is to find the smallest number of connections that is still capable of solving a given task with comparable accuracy, i.e. a simpler subnetwork. We achieve comparable performance for LeNet architectures on MNIST, and significantly higher parameter compression than state-of-the-art algorithms for VGG and ResNet architectures on CIFAR-10/100 and Tiny-ImageNet. Our approach also performs well for the two different optimizers considered—Adam and SGD. The algorithm is not designed to minimize FLOPs when considering current hardware and software implementations, although it performs reasonably when compared to the state of the art.

Files

S10994-024-06516-z.pdf
(pdf | 3.36 Mb)
- Embargo expired in 05-09-2024
License info not available