With the introduction of event-based cameras, such as the dynamic vision sensor (DVS), new opportunities have arisen for low-latency real-time visual data processing. Unlike traditional frame-based cameras that capture entire frames at fixed intervals, each pixel in an event-based camera operates asynchronously, generating an event whenever its brightness change exceeds a certain threshold. Although DVS sensors inherently surpass traditional frame-based cameras in capturing transient, high-speed phenomena, their performance bottleneck is usually located in their address event representation (AER) readout interfaces. The commonly used row-scanning synchronous AER, which encodes events in a full row at once, offers high throughput. However, this approach also introduces inherent delays that limit its use in applications requiring high temporal resolution. Conversely, while AER schemes based on asynchronous digital circuits surpass synchronous schemes in temporal resolution, their event-by-event transmission approach limits their overall throughput. This work proposes a novel high-speed asynchronous AER interface, leveraging spatiotemporal correlations in DVS event-based data, to optimize the tradeoff between temporal resolution and throughput. Supported by the recently proposed open-source asynchronous design toolkit (ACT) flow for asynchronous digital circuits, we propose an address fuser to be integrated into the hierarchical token ring (HTR) AER scheme. This address fuser creates a spatiotemporal window to exploit the inherent spatiotemporal correlations in DVS data. After verification at both switch- and transistor-level simulations, we benchmarked our design against the conventional HTR AER scheme using a representative set of input scenarios. Our design achieved 196% of the throughput for multi-event transmissions when all pixels were activated simultaneously, at the expense of an acceptable 18% latency increase for single-event transmissions with a 10-ns temporal window.

The creation of effective computational models that function within the power limitations of edge de- vices is an important research problem in the field of Artificial Intelligence (AI). While cutting-edge deep learning algorithms show promising results, they frequently need computing resources that are many orders of magnitude more than the available power and memory budgets for these devices. During the thesis, two unique learning algorithms (backpropagation and forward-forward) were developed and compared using the Teensy 4.1, a low-cost microcontroller board. This work seeks to bridge the gap between the necessary computing efficiency and the hardware’s restricted resources. By creating and analyzing these algorithms, with the Fashion MNIST dataset as a validation set, this thesis creates a baseline for AI efficiency on microcontrollers, with performance targets set at a mini- mum of 80% test accuracy. The microcontroller software, implemented in C++, is limited to using less than 512 kB RAM for all online training methods. In addition, the potential of transfer learning was also explored. Key performance parameters, including memory utilization, training and inference times, and accu- racy, were analyzed in a comparative study of the backpropagation and forward-forward algorithms. For each learning algorithm, several configurations were explored (such as topologies, and optimizers) to determine the most effective and efficient way for AI implementation on low-cost hardware. The key conclusions of this study reveal that backpropagation demonstrates superior performance in terms of both accuracy and computational efficiency. However, it requires more memory for storing variables, which may be a constraint in on-edge environments. Conversely, the forward-forward algorithm, while achieving lower accuracy, is more memory-efficient, making it a potential choice for less complex tasks or systems with severe RAM limitations. The application of transfer learning showed potential to accelerate the learning process and to improve the final accuracy, hinting at an effective strategy for deploying advanced AI models on resource-limited edge devices.

In the past decades, much progress has been made in the field of AI, and now many different algorithms exist that reach very high accuracies. Unfortunately, many of these algorithms are quite resource intensive, which makes them unavailable on low-cost devices. The aim of this thesis is to explore algorithms and neural network techniques suitable for implementation on FPGAs. While FPGAs provide almost complete control over all aspects of design, allowing for the development of high-performance systems, they have not gained widespread popularity in neural network development due to their limited accessibility compared to computers and microcontrollers. In the thesis, an inference-only 8-bit quantized neural net is designed, implemented and deployed on the Diligent Zedboard, and the performance is compared to similar networks on other devices. The thesis then focuses on two learning algorithms: Forward-Forward learning and Hebbian learning. It is shown how Forward-Forward can be seen as a way to apply Hebbian learning rules, and a simplified algorithm is proposed for use in a quantized system and implemented on an FPGA. Although the performance of the network is quite low, reaching only 90.5% on the MNIST dataset and 74.2% on Fashion MNIST, the results are promising enough to give ground for further research and show that even very simplified versions of the Forward-Forward algorithm are capable of learning. Moreover, it demonstrates that the Forward-Forward algorithm is suitable for FPGA implementation. Both implementations show that the processing speed of the FPGA implementations is much faster than that of similar network implementations on other devices.

Artificial Intelligence has become a dominant part of our lives, however, complex artificial intelligence models tend to use a lot of energy, computationally complex operations, and a lot of memory resources. Therefore, it excluded a whole class of hardware in its applicability. Namely, relatively resource-constrained low-cost hardware. This paper investigates learning methods that are potentially better suited for these types of devices: the forward-forward algorithm and Hebbian learning rules. The results are compared to backpropagation with equivalent network configurations, training hyperparameters and internal data types on different types of low-cost hardware. Backpropagation has consistently outperformed other algorithms in various tests. It exhibits higher accuracy, faster training, and faster inference compared to forward-forward models. However, forward-forward models can come close to matching backpropagation's performance, but they suffer from longer training times and decreased performance with multi-layer networks. Additionally, a poorly trained forward-forward model is sensitive to quantization, resulting in a significant drop in accuracy. On the other hand, forward-forward models offer the benefit of independently training each layer, allowing for more flexibility in optimizing the training process. Hebbian models were not found to be competitive, displaying performance below the required threshold. Smaller models for MCU and FPGA would likely perform even worse.

Olfactory learning in Drosophila larvae exemplifies efficient neural processing in a small-scale network with minimal power consumption. This system enables larvae to anticipate important outcomes based on new and familiar odor stimuli, a process crucial for survival and adaptation. Central to this learning mechanism is the olfactory pathway model, which embodies the principles of synaptic plasticity and associative learning through prediction error coding mediated by specific neuromodulating neurons in the mushroom body, like dopaminergic neurons. There is a pressing need to develop novel computational frameworks that capture the spatio-temporal processes while remaining compatible with the constraints of small-scale neural networks. These frameworks should draw inspiration from the biophysical properties of neurons within the olfactory pathway model, enabling accurate emulation of neural dynamics and efficient learning processes using spiking neural networks. This thesis proposes a framework based on a phenomenological conductance-based leaky integrate-and-fire (COBALIF) neuron model, inspired by the olfactory pathway model of Drosophila larvae. By first prototyping the spiking neural network in Intel's Lava Python-based framework, we validated the design on a neuron and system level for a neuromorphic hardware implementation. This was the foundation of a programmable, neuromorphic FPGA architecture capable of adaptive optimization, employed on a Zynq 7000 SoC FPGA. By implementing this architecture in a single-precision floating-point format, we model the real-time neural dynamics of the COBALIF neuron in one-tenth of a millisecond precision. Moreover, our FPGA implementation serves as a feasible prototype for deploying such biologically inspired neurons and their spatio-temporal dependencies in digital design, paving the way for scaling up to small-scale networks.

This thesis serves to finalise the bachelor graduation project on the topic of self-supervised federated learning, specifically the on-chip implementation of the algorithms. The goal of the project is to implement a self-supervised learning setup in a decentralised approach using Field-Programmable Gate Arrays (FPGAs) for the processing of data. In this thesis, we endeavour to illustrate the possibility of employing FPGAs to move the fairly compute-intensive self-supervised learning algorithms to the edge. We have developed a number of modules that can accelerate key algorithmic blocks that underlie the major bottlenecks of the classical application of the algorithms and showcase prospective results, which are extensively discussed afterwards to pave a clear path towards truly autonomous and efficient edge-intelligence.

Event-based cameras promise new opportunities for smart vision systems deployed at the edge. Contrary to their conventional frame-based counterparts, event-based cameras generate temporal light intensity changes as events on a per-pixel basis, enabling ultra-low latency with microsecond-scale temporal resolution, low power consumption at milliwatts level, and sparse information encoding where only dynamic objects trigger events, effectively excluding static background data. However, mainstream computer vision algorithms based on convolutional neural networks (CNNs) hardly exploit these advantages of event-based cameras. Recently, event graph neural networks (event-GNNs) have been proposed as the backbone for novel event-based vision algorithms. By treating events as graph data, GNNs are able to process events while preserving their spatiotemporal information and sparse characteristics. Further studies also revealed an event-driven computation workflow that translates an event stream into a dynamic, evolving graph, outlining a path toward low-latency event-based vision. Despite these promises, event-GNNs are still calling for dedicated hardware accelerators toward integrated solutions with real-time prediction latency and low power consumption for real-world edge intelligence. In this thesis, for the first time, we proposed an event-driven GNN accelerator for low-power, high-speed edge vision. Through hardware-algorithm co-design, an event-driven GNN model is adopted for deployment on an edge FPGA platform without prediction accuracy loss. We also pointed out two novel optimizations, edge-free storage and layer-parallel computation, to further decrease memory footprints and processing latency. The proposed accelerator is implemented on the Xilinx KV260 System-On-Module (SOM) platform containing an UltraScale+ MPSoC FPGA, and benchmarked on-board. Targeting a car recognition task based on the NCars dataset, our accelerator achieves a prediction accuracy of 87.8%. Meanwhile, operating with a 6.86W board-level system power, the accelerator reaches an average 16μs prediction latency per event and runs 9.2× faster than its software counterpart running on an NVIDIA RTX A6000 GPU platform. Therefore, our event-driven GNN accelerator efficiently allows for both real-time and microsecond-resolution event-based vision at the edge.

To support the spike propagates between neurons, neuromorphic computing systems always require a high-speed communication link. Meanwhile, spiking neural networks are event-driven so that the communication links normally exclude the clock signal and related blocks. This thesis aims to develop a self-timed off-chip interconnect system with ring topology that supports multi-point communication in neuromorphic computing systems. This interconnect system is implemented in high-level modeling with SystemC and involves the burst-mode two-wire protocol in point-to-point communication. In order to ensure the flexibility of the system, the distributed control system is involved. Further, the system can be configured with different numbers of chiplets to fulfill various spiking neural network structures. We also explore optimization methods, which is a bi-directional ring topology achieving the growth of throughput. Based on evaluation and simulation results, the interconnect system can achieve 4.302Gbps with the specific application scenario.

The growing interest in edge computing is driving the demand for more efficient deep learning models that fit into resource-constrained edge devices like Internet-of-Things (IoT) sensors. The challenging limitations of these devices in terms of size and power has given rise to the field of tinyML, focusing on enabling low-cost machine learning on edge devices. Up until recently, the work in this space was primarily focused on static inference scenarios. However, a prominent issue with this is that models cannot adapt post-deployment, leading to robustness issues with shifting data distributions or the introduction of new features in the data. However, at the edge, full on-device retraining, or communicating all new data to a central server, is infeasible: this necessitates the development of data-efficient learning algorithms to adapt locally and autonomously from streaming data. This challenge at the intersection of edge computing and data-efficient learning is currently an open challenge. In this thesis, we propose to solve this challenge with meta-learning. To clarify in which way the application of meta-learning is the most suitable for edge hardware, for the first time, a principled approach for meta-learning at the edge is outlined and investigated in three parts. The first part of this thesis details the selection of a suitable neural network architecture for few-shot learning over sequential data. By not being fixated on one architecture from the start, it is possible to explore different approaches to learning over sequences of temporal data, leading to the identification of the most effective architecture for generalizing from limited temporal examples. The quantitatively evaluated architectures are a recurrent neural network (RNN), a gated recurrent unit (GRU), a long-short-term memory (LSTM) and a temporal convolutional network (TCN). We show that TCNs outperform all architectures, while GRUs and LSTMs have a lower activation memory requirement. However, the latter require a linearly increasing number of multiplications with input sequence length, while it scales logarithmically for TCNs. Our results show that TCNs therefore provide the most favorable trade-off for low-cost temporal feature extraction at the edge. The second part of the thesis focuses on the algorithmic developments of the few-shot learning setup. Building on recent results from machine learning research, we highlight how meta-learning techniques primarily rely on learning high-quality features that generalize well. Taking into account hardware-driven considerations such as memory and compute overheads and through detailed quantitative analyses, we demonstrate that the best performance-cost trade-off is reached with a simple supervised pre-training scheme, where on-chip learning is performed by comparing the outputs of a TCN-based feature extractor with Manhattan distance. We also analyze the impact of quantization on this trade-off and, accordingly, we select a scheme with 4-bit logarithmic weights and 4-bit unsigned activations...

Motivated by the desire to bring intelligent processing at the Edge, enabling online learning on resource- and latency-constrained embedded devices has become increasingly appealing, as it has the potential to tackle a wide range of challenges: on the one hand, it can deal with on-the-fly adaptation to fast sensor-generated streams of data under changing environments and on the other hand, it can address a variety of challenges associated with offline training in the cloud, such as incurred energy consumption of sensor data transfers and extra memory storage for the training samples, but also data privacy and security concerns. Concurrently, maintaining low-latency and power-efficient inference is paramount for edge AI computing systems, and thus learning/adapting online with minimal incurred overhead is crucial. In this work, we propose EON-1, an Edge ONline Learning SCNN (Spiking Convolutional Neural Network) processor with 1-bit synaptic weights, 1-spike per neuron and 1-neuron updated per input, which we have benchmarked for both ASIC and FPGA platforms. Our key contribution is proposing a binary and stochastic SDTP rule which, benchmarked in an ASIC node, achieves less than 1% energy overhead for inference. To our knowledge, our solution incurs the least energy overhead for inference, compared to state-of-the-art solutions, showing a better efficiency by at least a factor of 10x. We also report 94% and 77.65% accuracy on the MNIST and Fashion-MNIST classification tasks, and we achieve 0.09pJ/SOP and 1.5pJ/SOP energy efficiency during inference and learning, respectively. We extend our solution to demonstrate a practical use-case of performing inference in real-time UHD videos while coping with streaming data and we showcase 60 FPS UHD video processing.

Recent trends in machine learning (ML) have placed a strong emphasis on power- and resource-efficient neural networks, as well as the development of neural networks on edge devices. Spiking neural net-works (SNNs), due to their event-based nature, are one of the most promising types of neural networks for low-power applications. To accelerate and ease the deployment of SNNs on edge devices, this thesis presents a configurable digital neuromorphic hardware generator for heterogeneous computing that is capable of generating resource-efficient SNN processing cores. The proposed hardware generator is de-veloped using SpinalHDL, a high-level hardware description language (HDL), which provides a high level of flexibility in hardware generation. Our generator supports the configuration on various parameters and is capable of generating a tree-structured multi-core architecture of heterogeneous cores. The generator is deployed in a sensor-fusion hand-gesture classification use case, for which the configurability of our hardware generator is a key enabler.

Voice activity detection (VAD) is the prevailing approach to extracting meaningful speech information from the pervasive noise found in the physical environment. Presently, deep neural networks (DNN) are widely employed as the classifier component in Voice Activity Detection (VAD) systems. However, conventional deep neural networks, like fully connected (FC) deep neural networks, encounter the challenge of excessive computational complexity. This heightened complexity can result in diminished computing efficiency, unnecessary utilization of hardware resources, and redundant power consumption. To address the inefficiency issue from computational complexity, this study introduces a novel neural network architecture named DeltaFC. This architecture attains an operation time latency of less than 1 ms for each 30ms voice segment, resulting in a 54% reduction in latency compared to the baseline fully connected (FC) model. In software design, this study tackles the issue by compressing and encoding time-series information using the Delta algorithm, with the objective of introducing temporal sparsity. Based on the software results, the neural network surpasses both the baseline fully connected (FC) and LSTM models in AUC (area under the curve), with accuracy at a lightweight parameter scale. In hardware design, this study reproduces the neural network software design into FPGA hardware RTL design, implementing a lightweight digital IP core. This digital IP core accelerates neural network operations in hardware by the deployment of Delta and CSR algorithms. Compared with not introducing temporal sparsity, the computing efficiency increases by approximately 85% with 0.5% loss in accuracy. This substantiates that within the domain of lightweight neural networks containing fewer than 30,000 parameters, the DeltaFC network proposed in this study is more suitable for Voice Activity Detection (VAD) when compared to fully connected (FC), LSTM, and other baseline network architectures.

This report serves to finalize the bachelor graduation project on the topic of self-supervised federated learning, specifically the implementation of the algorithms in Python. The goal of the project is to implement a self-supervised learning setup in a decentralized approach using Field-Programmable Gate Arrays (FPGAs) for the processing of data. This serves as a proof of concept that decentralized machine learning on unlabeled data using FPGAs is possible. Multiple algorithms based on the literature were considered to allow for a low-profile learning setup, with simplifications done to be able to reduce the compute required. The results are promising: scaled-down models that can run on an FPGA show that self-supervised learning functions as expected from the theory. By decentralizing the computations increases in performance are possible in favorable conditions. The authors hope that the concept of self-supervised federated learning can be employed to FPGAs on a larger scale to help in the processing of the abundant yet underutilized unlabeled data present at the edges of information networks.

This report serves to finalize the bachelor graduation project on the topic of self-supervised federated learning, specifically the implementation of the algorithms in Python. The goal of the project is to implement a self-supervised learning setup in a decentralized approach using Field-Programmable Gate Arrays (FPGAs) for the processing of data. This serves as a proof of concept that decentralized machine learning on unlabeled data using FPGAs is possible. Multiple algorithms based on the literature were considered to allow for a low-profile learning setup, with simplifications done to be able to reduce the compute required. The results are promising: scaled-down models that can run on an FPGA show that self-supervised learning functions as expected from the theory. By decentralizing the computations increases in performance are possible in favorable conditions. The authors hope that the concept of self-supervised federated learning can be employed to FPGAs on a larger scale to help in the processing of the abundant yet underutilized unlabeled data present at the edges of information networks.