The focus of this project was the research and development of an affective social touch wearable. AffectiveAir uses pneumatic actuation on the shoulder to convey a library of haptic sensations. The goal was to overcome physical limitations in potential digital communication contexts, by a non-verbal, tactile, and possibly intimate social touch using pneumatic actuation. This would offer a way for individuals to connect, much like in face-to-face interactions The embodiment of the prototype is the result of an iterative design process based on literature research, benchmarks, and user tests. The wearable actuator, designed to be worn on the back of the shoulder, is secured using an elastic band that attaches to the wearer’s pants. A custom-designed, thin polyurethane airbag with an integrated nozzle provides a soft, textile-integrated actuator solution. An external pneumatic control system controls the airflow, allowing for inflation under 1000- and deflation under 100 milliseconds. Effects of the actuator are monitored using an air pressure sensor and force sensitive resistor, offering controlled feedback and data logs for prototyping and user research. Multiple user tests functioned as a means to explore and verify new designs throughout the project. Initial tests identified the shoulder as an effective and acceptable location for the feedback, and an airbag of 40x40mm was determined to provide the best balance between intensity and comfort for this specific location. Subsequent tests determined optimal pressure levels for pneumatic haptics on the shoulder and evaluated user responses to various pneumatic patterns. Results revealed that it was challenging for participants to distinguish pressure levels from each other within the 0 to 500 mbar range. However, there were promising results in the general identification of increasing or decreasing pneumatic patterns using three pressure levels of 75, 200 and 500 mbar. ‘TripleShort’ received the highest identification rating at 85%, while ‘short staircase down’ received the lowest rating at 44%. Patterns with a prolonged high pressure level at the end such as ‘heartbeat forward’ and ‘long staircase up’ received a slighly higher comfort and pleasantness rating compared to others, where the latter was rated the least exciting at lower speed. Overall ratings remained close to the median across tests with variable speed or pressure levels. General user feedback on the prototype and haptic experience was positive, with participants noting the novelty and the sensation’s occasional resemblance to intimate human touch. Feedback from the interview also notes how context and social relationship status closely relates to the acceptability of receiving such feedback. The ergonomics of the prototype were considered satisfactory, allowing for adjustment to fit any body size and position the feedback on the back of the participant’s shoulder. In conclusion, AffectiveAir demonstrated the potential for affective touch through a pneumatic shoulder wearable. It offers a library of identifiable and characterized pneumatic patterns and the ability to extend the possible actuations with adjustable parameters. The project’s outcomes suggest promising potential for further research using affective pneumatic haptics, applicable not only in mediated communication but also in other possible areas such as gaming and navigation.

Emotion Recognition is one of the vastly studied areas of affective computing. Attempts have been made to design emotion recognition systems for everyday settings. The ubiquitous nature of Intelligent voice assistants (IVAs) in households, make them a great anchor for the introduction of emotion recognition technology to consumers. The existing systems lack such pipelines and rely on dictionary-based architectures in their design. Further, these systems lack conversational properties and are merely an extension of information retrieval engines. In this setting, we propose to introduce and develop emotion recognition pipelines that are suited to the interactions, common with these IVAs. To augment the existing emotion recognition pipelines which rely on audio information, we look at physiological information derived from wearables. Our proposed model uses multimodal embeddings with a Siamese Network to achieve the task of emotion recognition from a few samples. Physiological signals of blood volume pulse (BVP) and electrodermal activity (EDA) are used as additional input embeddings to two audio embeddings arising from the speech samples. We employ the state-of-the-art training schedules for Siamese Networks, which use a very limited amount of training on support datasets via sample pair comparisons. The performance of the model is evaluated using weighted binary accuracy and f1 scores. The proposed model is applied on two datasets that denote two unique experimental settings - the K-EmoCon dataset and RECOLA dataset. We demonstrate an improvement in the state-of-the-art accuracy with the K-EmoCon dataset with accuracies of 63.97% and 66.91% on arousal and valence dimensions respectively. Further, on the RECOLA dataset, the model performs moderately well with 53.81% and 53.87% respectively for arousal and valence dimensions. In addition to this, we present a study of the effects of variation of available support set for training from the dataset. We make some salient observations for these experiments across individual participants and also identify how the label distributions affect the performance of the model. Further, we investigate the impact of real-world noise samples from the DEMAND dataset on the two datasets. We observe that the proposed model is robust and performs sustainingly well even in the presence of imputed noise.

Sleep is a natural state of our mind and body during which our muscles heal and our memories are consolidated. It is such a habitual phenomenon that we have been viewing it as another ordinary task in our day-to-day life. However, owing to the current fast-paced, technology-driven generation, we are letting ourselves be sleep-deprived, giving way to serious health concerns such as depression, insomnia, restlessness, apnea and Alzheimer’s. Polysomnography (PSG) studies are used for diagnosing and treating sleep-related disorders. Although the PSG studies are considered as the gold standard, they are obtrusive and do not allow for long-term monitoring. Various wearables have been manufactured to help people monitor their sleep-health. However, these devices have been shown to be inaccurate. The ubiquitous sensor technology employed by the wearables provides large volumes of data, recorded in the most natural setting of the user. There is an opportunity to make use of the highly available sensor data to model a sleep scoring system that could help individuals monitor their sleep-health from the comfort of their home. In this thesis, we aim to alleviate this problem by attempting to bridge the gap between the highly accurate but obtrusive medical diagnosis (PSG) and the non-intrusive yet inaccurate wearables. In this work, we propose DeepSleep, a deep neural net-based sleep classification model using an unobtrusive BCG-based heart sensor signal. Our proposed model’s architecture uses the combination of CNN and LSTM layers to perform self-feature extraction and sequential learning respectively. We show that our model can classify sleep stages with a mean f1-score of 74% using the BCG signal. We employ a 2-phase training strategy to build a pre-trained model to tackle the limited dataset size and test the transferability of the model on other types of heart-signal. With an average classification accuracy of 82% and 63% using ECG and PPG based heart signal respectively, we show that our pre-trained model can be used in the transfer learning setting as well. Lastly, with the help of a user study of 16 subjects, we show that the objective sleep quality metrics correlate with the perceived sleep quality reported by the subjects with a correlation score of 푟 = 0.43. Although our proposed model’s performance is not yet comparable to the medical standards, we show that it is possible to monitor our sleep-health using the wearable signals with the least domain knowledge and preprocessing techniques. The prediction and performance of our DeepSleep model show that it is able to learn the biological rules of sleep wherein it always follows a Deep or REM stage with a transitional Light stage. Our model treats the classification problem sequentially, thus, identifying important sleep parameters like the onset of sleep cycles and time spent in different sleep stages which are time-dependent factors. Furthermore, our user study, conducted using the SATED questionnaire, provides an insight into the difference in the user’s perceived sleep quality and model’s estimation. It shows that an automated classification system needs to incorporate various external factors such as environmental and ambient conditions to be able to strongly correlate with the perceived or subjective quality. We further discuss the future research gaps and opportunities that could improve the model’s performance and also extend it to other domains like irregular heart-beat and apnea detection. We consider this work to be a starting point for research into sleep and heart health using non-intrusive wearable sensors and deep neural network-based architectures.

Fine-grained emotion recognition is the process of automatically identifying the emotions of users at a fine granularity level, typically in the time intervals of 0.5s to 4s according to the expected duration of emotions. Previous work mainly focused on developing algorithms to recognize only one emotion for a video based on the user feedback after watching the video. These methods are known as post-stimuli emotion recognition. Compared to post-stimuli emotion recognition, fine-grained emotion recognition can provide segment-by-segment prediction results, making it possible to capture the temporal dynamics of users’ emotions when watching videos. The recognition result it provides can be aligned with the video content and tell us which specific content in the video evokes which emotions. Most of the previous works on fine-grained emotion recognition require fine-grained emotion labels to train the recognition algorithm. However, the experiments to collect these fine-grained emotion labels are usually costly and time-consuming. Thus, this thesis focuses on investigating whether we can accurately predict the emotions of users at a fine granularity level with only a limited amount of emotion ground truth labels for training. We start our technical contribution in Chapter 3 by building up the baseline methods which are trained using fine-grained emotion labels. This can help us understand how accurate the recognition can be if we take advantage of the fine-grained emotion labels. We propose a correlation-based emotion recognition algorithm (CorrNet) to recognize the valence and arousal (V-A) of each instance (fine-grained segment of signals) using physiological signals. CorrNet extracts features both inside each fine-grained signal segment (instance) and between different instances for the same video stimuli (correlation-based features). We found out that, compared to sequential learning, correlation-based instance learning offers advantages of higher recognition accuracy, less overfitting and less computational complexity. Compared to collecting fine-grained emotion labels, it is easier to collect only one emotion label after the user watched that stimulus (i.e., the post-stimuli emotion labels). Therefore, in the second technical chapter (Chapter 4) of the thesis, we investigate whether the emotions can be recognized at a fine granularity level by training with only post-stimuli emotion labels (i.e., labels users annotated after watching videos), and propose an Emotion recognition algorithm based on Deep Multiple Instance Learning (EDMIL). EDMIL recognizes fine- grained valence and arousal (V-A) labels by identifying which instances represent the post-stimuli V-A annotated by users after watching the videos. Instead of fully-supervised training, the instances are weakly-supervised by the post-stimuli labels in the training stage. Our experiments show that weakly supervised learning can reduce overfitting caused by the temporal mismatch between fine-grained annotations and input signals. Although the weakly-supervised learning algorithm developed in Chapter 4 can obtain accurate recognition results with only few annotations, it can only identify the annotated (post-stimuli) emotion from the baseline emotion (e.g., neutral) because only post-stimuli labels are used for training. The non-annotated emotions are all categorized as part of the baseline. To overcome this, in Chapter 5, we propose an Emotion recognition algorithm based on Deep Siamese Networks (EmoDSN). EmoDSN recognizes fine-grained valence and arousal (V-A) labels by maximizing the distance metric between signal segments with different V-A labels. According to the experiments we run in this chapter, EmoDSN achieves promising results by using only 5 shots (5 samples in each emotion category) of training data. Reflecting on the achievements reported in this thesis, we conclude that the fully-supervised algorithm (Chapter 3) can result in more accurate fine-grained emotion recognition results if the annotation quantity is sufficient. The weakly-supervised learning method (Chapter 4) can result in better recognition results at the instance level compared to fully-supervised methods. We also found that the weakly-supervised learning methods can perform the best if users annotate their most salient, but short emotions or their overall and longer-duration (i.e., persisting) emotions. The few-shot learning method (Chapter 5) can obtain more emotion categories (more than the weakly-supervised learning) by using less amount of samples for training (better than the fully-supervised learning). However, the limitation of it is that accurate recognition results can only be achieved by a subject-dependent model.@en