Active dendrites for continual learning in a time-to-first-spike spiking neural network architecture

Abstract

Spiking neural networks (SNNs) are a new generation of neural networks aiming at reducing the power consumption of conventional artificial intelligence systems by mimicking the behaviour of biological neurons found in the human brain. To achieve this goal, SNNs mimic the propagation of information observed in biological neurons through the use of discrete events known as spikes. Historically, different theories have been proposed to explain how information is encoded into these spike events. One such theory is time-to-first-spike (TTFS) coding, which offers valuable opportunities for low-power and low-latency hardware implementations.

Nonetheless, networks of spiking neurons still miss a characteristic of learning observed in human beings. Specifically, they are unable to learn different tasks in a sequential fashion without incurring the problem of catastrophic forgetting. Indeed, while these networks achieve state-of-the-art results in a vast number of problems, they require full retraining of the network as new tasks need to be learned. This problem not only highlights a difference from biological systems, but also limits the applicability of such systems in environments which require adaptation to new tasks.

Currently, there exists no hardware that is capable of mitigating the problem of catastrophic forgetting while leveraging the low-power and low-latency opportunities offered by TTFS coding. To overcome this research gap, we conducted a literature review of proposed solutions to the problem of catastrophic forgetting in both the artificial and spiking neural network domains. The aim of this review is to uncover biologically inspired solutions to the problem of catastrophic forgetting which could be applied to TTFS-encoded spiking networks. Furthermore, to implement a digital hardware accelerator capable of incorporating the requirements of the selected solution, we summarized key architectures for event-based SNNs.

This thesis presents a novel neural model based on a spike response model (SRM) with a Rel-PSP kernel, which is enhanced with active dendrites. The proposed solution successfully mitigates the problem of catastrophic forgetting in a typical continual learning setup, in which the network is trained over different tasks in a sequential fashion, i.e. one task after the other. Additionally, a digital hardware architecture was designed that implements the proposed solution on a Xilinx Zynq-7020 SoC FPGA.

Our solution is capable of learning the first five digits of the N-MNIST dataset in a sequential fashion, resulting in a final average accuracy of 100% across all tasks. Conversely, the same model without active dendrites achieves an accuracy of only 23%, which is close to random guessing, thereby demonstrating a successful mitigation of catastrophic forgetting with the proposed solution. Additionally, our digital hardware implementation is capable of classifying a sample image of the dataset in an average time of 117 μs while consuming 232 mW at a clock frequency of 125 MHz. The proposed architecture uses 74% of the LUTs, 28% of the FFs, and 32% of the BRAM available in the FPGA.