Side-Channel Attacks, are a prominent type of attacks, used to break cryptographic implementations on a computing system. They are based on information "leaked" by the hardware of a computing system, rather than the encryption algorithm itself. Recent studies showed that Side-Channel Attacks can be performed using Deep Learning models. In this study, we examine the performance of Convolutional Neural Networks, on four different datasets of side- channel data and we compare our models with conventional Machine Learning algorithms and a CNN model from literature. We found that CNNs have the potential to achieve high accuracy performance (99.8%), although their capacity is heavily influenced by the use case. We also found that certain Machine Learning algorithms can outperform CNNs in certain cases, leaving an open debate on the performance gains of the latter.

Images possess the ability to convey a wide range of emotions, and extracting affective information from images is crucial for affect prediction systems. This process can be achieved through the application of machine learning algorithms. Categorical Emotion States (CES) and Dimensional Emotion Space (DES) are two typical models used for representing emotions. Moreover, the development of a mapping schema between these representations can contribute to benefit the research in AI and psychology. Consequently, this study focuses on investigating the feasibility of translating emotions from DES to CES. To accomplish this goal, relevant databases are identified and combined as the training data, and machine learning models, namely Naive Bayes, K-nearest Neighbors, and Decision Tree are employed to perform the classification task. The results indicate the superior performance of the K-nearest Neighbors classifier, exhibiting higher mean accuracy (60%) and low standard deviation (0.004) among all implemented classifiers. Overall, the translation from DES to CES offers several advantages, including a simplified and interpretable representation of emotions, as well as the provision of a common language for discussing and expressing emotions.

This report describes the implementation of a custom streaming solution from an IP camera to a web browser. The system aims to both provide live video and Video on Demand. This will be used to monitor premature babies in incubators.

Surgical navigation is a tool that surgeons rely on everyday to perform accurate surgeries all over the world. However, this technology requires good hand-eye coordination and a high level of concentration. HoloNav is a project that inquires to see if using the HoloLens and augmented reality can replace the current surgical navigation methods. To do so, the HoloLens must be able to identify the patient and the location of the surgery instruments, which uses optical reflective spheres. This study focuses on using the grayscale cameras of the HoloLens and a deep learning algorithm YOLOv5 to test if it is possible to precisely detect optical reflective spheres. 3 models were trained with two different data sets, where the results show that the model trained on a data set would perform well on the validation set. However, they would perform far worse when exposed to a data set it was not trained on.

Using Recommender Systems with Evolutionary Algorithms is an extremely niche domain. It holds the key to enabling new user interaction designs, where users can effectively configure their experience with a Recommender System. This thesis answers important questions about the scientific aspects of its application to large-scale data through a rigorous experimental design. We use one of the largest publicly accessible music listening histories dataset to analyse if the methodology works well for large real-world tasks. The dataset has been used to simulate various real-world scenarios for the experimental design. The methodology fuses the recommendations generated by an unspecified number of recommenders. In this study, we have used three recommenders, which have specialised goals in terms of user-centric metrics. We use three different Evolutionary Algorithms to analyse the capability of different EA strategies for generating a near-optimal set of trade-offs on user-centric metrics. We have performed elaborate qualitative and quantitative analyses of the system to understand how various aspects of the system affect the final set of solutions.

High-level music classification tasks such as automatic music mood annotation impose several challenges, both from a psychological and a machine learning point of view. Ground truth labels for these tasks at hand are hard to define due to the abstract and aesthetic nature of the data, being largely dependent on human psychology and perception. Such labels, however, are required in training and validation sets when traditional machine learning methods are used. Furthermore, due to copyright restrictions which prevent the sharing of commercial music audio, such classifiers have to work with pre-computed music audio features which are known to be somewhat unstable. Due to the challenges inherent to high-level music classification, the following questions arise: do high-level music classifiers actually perform as well as we believe? And can we trust their output to be a solid foundation for future research? This work analyzes the performance of high-level music classifiers using metrics based on label stability, label agreement and distributional differences, all of which do not depend on any problematic ground truth labels, on a dataset which combines data from AcousticBrainz and Spotify. Unexpected patterns in classifier outputs are uncovered, indicating that these outputs should not be taken as absolute truth and do not form a solid foundation for further research. The improvement of these high-level music classifiers is a multidisciplinary effort for which better evaluation methods are required. To this end, several approaches for more comprehensive classifier testing are presented, based on best practices in psychology and software testing. These approaches are not constrained to the field of Music Information Retrieval and can be applied to evaluate classifiers in other domains as well.

Sleep is a natural state of our mind and body during which our muscles heal and our memories are consolidated. It is such a habitual phenomenon that we have been viewing it as another ordinary task in our day-to-day life. However, owing to the current fast-paced, technology-driven generation, we are letting ourselves be sleep-deprived, giving way to serious health concerns such as depression, insomnia, restlessness, apnea and Alzheimer’s. Polysomnography (PSG) studies are used for diagnosing and treating sleep-related disorders. Although the PSG studies are considered as the gold standard, they are obtrusive and do not allow for long-term monitoring. Various wearables have been manufactured to help people monitor their sleep-health. However, these devices have been shown to be inaccurate. The ubiquitous sensor technology employed by the wearables provides large volumes of data, recorded in the most natural setting of the user. There is an opportunity to make use of the highly available sensor data to model a sleep scoring system that could help individuals monitor their sleep-health from the comfort of their home. In this thesis, we aim to alleviate this problem by attempting to bridge the gap between the highly accurate but obtrusive medical diagnosis (PSG) and the non-intrusive yet inaccurate wearables. In this work, we propose DeepSleep, a deep neural net-based sleep classification model using an unobtrusive BCG-based heart sensor signal. Our proposed model’s architecture uses the combination of CNN and LSTM layers to perform self-feature extraction and sequential learning respectively. We show that our model can classify sleep stages with a mean f1-score of 74% using the BCG signal. We employ a 2-phase training strategy to build a pre-trained model to tackle the limited dataset size and test the transferability of the model on other types of heart-signal. With an average classification accuracy of 82% and 63% using ECG and PPG based heart signal respectively, we show that our pre-trained model can be used in the transfer learning setting as well. Lastly, with the help of a user study of 16 subjects, we show that the objective sleep quality metrics correlate with the perceived sleep quality reported by the subjects with a correlation score of 푟 = 0.43. Although our proposed model’s performance is not yet comparable to the medical standards, we show that it is possible to monitor our sleep-health using the wearable signals with the least domain knowledge and preprocessing techniques. The prediction and performance of our DeepSleep model show that it is able to learn the biological rules of sleep wherein it always follows a Deep or REM stage with a transitional Light stage. Our model treats the classification problem sequentially, thus, identifying important sleep parameters like the onset of sleep cycles and time spent in different sleep stages which are time-dependent factors. Furthermore, our user study, conducted using the SATED questionnaire, provides an insight into the difference in the user’s perceived sleep quality and model’s estimation. It shows that an automated classification system needs to incorporate various external factors such as environmental and ambient conditions to be able to strongly correlate with the perceived or subjective quality. We further discuss the future research gaps and opportunities that could improve the model’s performance and also extend it to other domains like irregular heart-beat and apnea detection. We consider this work to be a starting point for research into sleep and heart health using non-intrusive wearable sensors and deep neural network-based architectures.

Music consumption has skyrocketed in the past few years with advancements in internet and streaming technologies. This has resulted in the rapid development of the inter-disciplinary field of Music Information Retrieval (MIR), which develops automatic methods to efficiently and effectively access the wealth of musical content. In general, research in MIR has focused on tasks like semantic filtering, annotation, classification and search. Observing the evolution of MIR over the years, research in this field has been focusing on “what music is” and in this thesis we move towards building tools that can analyse “what music does” to the listener. There is little research on building systems that analyse how music affects the listener or how people use music to suit their needs. In this thesis, we propose methods that push the boundaries of this perspective. The first major part of the thesis focuses on detecting high-level events in music tracks. Research on event detection in music has been restricted to detecting low-level events viz., onsets. There is also an abundance of literature on music auto-tagging, where researchers have focused on adding semantic tags to short music snippets. However, we look at the problem of event detection from a different perspective and turn to social music sharing platform – SoundCloud to understand what events are of importance to the actual listeners. Using a case-study in Electronic Dance Music (EDM), we design an approach to detect high-level events in music. The high-level events in our case-study have a certain impact on the listeners causing them to comment about these events on SoundCloud. Through successful experiments, we demonstrate how these high-level events can be detected efficiently using freely available but noisy user comments. The results of this approach inspired us for further research to investigate other tasks that can give us more insight into how music affects the listener. The second major part of the thesis concerns identifying music that can support different common activities – working, studying, relaxing, working out etc. A certain type of music is suitable for enabling listeners to perform a certain task. We first investigate what activities are important from a listeners’ perspective, for which music is sought, through a data-driven experiment on YouTube. After illustrating how existing music metadata like genre, instrument is insufficient, we propose a method that can successfully classify music based on the activity categories. An important insight from our experiments is that dividing the music track into short frames is not an effective method of feature extraction for activity-based music classification. This task requires a longer time window for feature extraction. Additionally, presence of high-level events like drop can affect the classification performance. After successful validation of our idea on activity-based music classification, we went on to investigate what can potentially distract a listener while doing a task. For this, we gathered valuable input from users of Amazon Mechanical Turk (AMT) on what musical characteristics distract them while doing their tasks. Based on this input, we built a system that can automatically detect a derail moment in a given music track, where the listener could potentially get distracted (derailed). Though this task seems to have a likely subjective component, we demonstrated that there are universal aspects to it as well. Through a literature survey and computational experiments, we demonstrate that we can automatically detect a derail moment. Throughout the thesis, we also stress on the importance of crowdsourcing platforms like AMT and social media sharing platforms like SoundCloud, and YouTube in understanding the user’s requirements and gathering data. We believe that our proposed methods and their outcomes will encourage future researchers to focus on this breed of MIR tasks, where the focus is on how music affects the listener. We also hope that the insights gained through this thesis will inspire designers and developers to build novel user interfaces to enable effective access of music. @en

Machine learning (ML) has become a core technology for many real-world applications. Modern ML models are applied to unprecedentedly complex and difficult challenges, including very large and subjective problems. For instance, applications towards multimedia understanding have been advanced substantially. Here, it is already prevalent that cultural/artistic objects such as music and videos are analyzed and served to users according to their preference, enabled throughML techniques. One of the most recent breakthroughs in ML is Deep Learning (DL), which has been immensely adopted to tackle such complex problems. DL allows for higher learning capacity, making end-to-end learning possible, which reduces the need for substantial engineering effort, while achieving high effectiveness. At the same time, this also makes DL models more complex than conventional ML models. Reports in several domains indicate that such more complex ML models may have potentially critical hidden problems: various biases embedded in the training data can emerge in the prediction, extremely sensitive models can make unaccountable mistakes. Furthermore, the black-box nature of the DL models hinders the interpretation of the mechanisms behind them. Such unexpected drawbacks result in a significant impact on the trustworthiness of the systems in which the ML models are equipped as the core apparatus. In this thesis, a series of studies investigates aspects of trustworthiness for complex ML applications, namely the reliability and explainability. Specifically, we focus on music as the primary domain of interest, considering its complexity and subjectivity. Due to this nature of music, ML models for music are necessarily complex for achieving meaningful effectiveness. As such, the reliability and explainability of music ML models are crucial in the field.@en

Fine-grained emotion recognition is the process of automatically identifying the emotions of users at a fine granularity level, typically in the time intervals of 0.5s to 4s according to the expected duration of emotions. Previous work mainly focused on developing algorithms to recognize only one emotion for a video based on the user feedback after watching the video. These methods are known as post-stimuli emotion recognition. Compared to post-stimuli emotion recognition, fine-grained emotion recognition can provide segment-by-segment prediction results, making it possible to capture the temporal dynamics of users’ emotions when watching videos. The recognition result it provides can be aligned with the video content and tell us which specific content in the video evokes which emotions. Most of the previous works on fine-grained emotion recognition require fine-grained emotion labels to train the recognition algorithm. However, the experiments to collect these fine-grained emotion labels are usually costly and time-consuming. Thus, this thesis focuses on investigating whether we can accurately predict the emotions of users at a fine granularity level with only a limited amount of emotion ground truth labels for training. We start our technical contribution in Chapter 3 by building up the baseline methods which are trained using fine-grained emotion labels. This can help us understand how accurate the recognition can be if we take advantage of the fine-grained emotion labels. We propose a correlation-based emotion recognition algorithm (CorrNet) to recognize the valence and arousal (V-A) of each instance (fine-grained segment of signals) using physiological signals. CorrNet extracts features both inside each fine-grained signal segment (instance) and between different instances for the same video stimuli (correlation-based features). We found out that, compared to sequential learning, correlation-based instance learning offers advantages of higher recognition accuracy, less overfitting and less computational complexity. Compared to collecting fine-grained emotion labels, it is easier to collect only one emotion label after the user watched that stimulus (i.e., the post-stimuli emotion labels). Therefore, in the second technical chapter (Chapter 4) of the thesis, we investigate whether the emotions can be recognized at a fine granularity level by training with only post-stimuli emotion labels (i.e., labels users annotated after watching videos), and propose an Emotion recognition algorithm based on Deep Multiple Instance Learning (EDMIL). EDMIL recognizes fine- grained valence and arousal (V-A) labels by identifying which instances represent the post-stimuli V-A annotated by users after watching the videos. Instead of fully-supervised training, the instances are weakly-supervised by the post-stimuli labels in the training stage. Our experiments show that weakly supervised learning can reduce overfitting caused by the temporal mismatch between fine-grained annotations and input signals. Although the weakly-supervised learning algorithm developed in Chapter 4 can obtain accurate recognition results with only few annotations, it can only identify the annotated (post-stimuli) emotion from the baseline emotion (e.g., neutral) because only post-stimuli labels are used for training. The non-annotated emotions are all categorized as part of the baseline. To overcome this, in Chapter 5, we propose an Emotion recognition algorithm based on Deep Siamese Networks (EmoDSN). EmoDSN recognizes fine-grained valence and arousal (V-A) labels by maximizing the distance metric between signal segments with different V-A labels. According to the experiments we run in this chapter, EmoDSN achieves promising results by using only 5 shots (5 samples in each emotion category) of training data. Reflecting on the achievements reported in this thesis, we conclude that the fully-supervised algorithm (Chapter 3) can result in more accurate fine-grained emotion recognition results if the annotation quantity is sufficient. The weakly-supervised learning method (Chapter 4) can result in better recognition results at the instance level compared to fully-supervised methods. We also found that the weakly-supervised learning methods can perform the best if users annotate their most salient, but short emotions or their overall and longer-duration (i.e., persisting) emotions. The few-shot learning method (Chapter 5) can obtain more emotion categories (more than the weakly-supervised learning) by using less amount of samples for training (better than the fully-supervised learning). However, the limitation of it is that accurate recognition results can only be achieved by a subject-dependent model.@en