Person | TU Delft Repository

Classification of Covert Vowels in Spanish and Dutch

What do brain signals say about inner speech?

Master thesis (2023) - I. Kyriazis, A.C. Schouten, O.E. Scharenborg, A. Seth, Y.B. Eisma

Patients with neuromuscular diseases that are unable to speak, but whose cognitive ability has been maintained, can be benefited from Brain Computer Interfaces (BCIs). The decoding of inner (covert) speech from EEGs consists of one of the state of the art methods that aim to tack ...

Patients with neuromuscular diseases that are unable to speak, but whose cognitive ability has been maintained, can be benefited from Brain Computer Interfaces (BCIs). The decoding of inner (covert) speech from EEGs consists of one of the state of the art methods that aim to tackle this issue. High variability between subjects, as well as low signal to noise ratio (SNR) undermine the methods used, and introduce the need for computer assisted solutions. Thus, machine learning models as well as large amounts of recorded data are required to design effective algorithms and produce substantial results. In this study, covert vowel classification was performed in a systematic way, by making use of two openly shared databases from literature; the Coretto database, that contains EEG recordings of native Spanish speakers, and the DAIS dataset, which includes EEG recordings of native Dutch speakers. Six classifiers were initially selected to perform 5-class classification: a Random Forest (RF), a k Nearest Neighbours (kNN), a Gaussian Naive Bayers (GNB), a Deep Convolutional Neural Network (DCNN), a Shallow Convolutional Neural Network (SCNN) and a Long Short Term Memory Recurrent Neural Network (LSTM). The DCNN outperformed the other methods, with average intra-subject accuracies of 35% for Coretto and 39% for DAIS (chance level 20%). Afterwards, an Overt versus Covert trials experiment was implemented, to test the limits of overt speech decoding from EEGs. The overt result was slightly higher than covert, with an intra-subject average value of 37.8% for Coretto and 40.5% for DAIS (chance level 20%). Finally, binary classification was performed to identify those pairs of vowels that can be classified more efficiently. Vowels /a/ and /u/ seemed to perform better in average in both datasets (average of 64.8% for Coretto and 64.4% for DAIS with a chance accuracy of 50%). Future work should focus on identifying the useful parts of the EEG recordings, increasing the SNR and the resolution of the electrodes, and defining the most appropriate dictionaries of words/vowels for a BCI. Also, more studies should follow systematic ways of comparisons between datasets, to obtain less ambiguous insights and lead this field to improvements.

Perceptions of Artificial Social Agents

The cultural similarities and differences between Dutch and Chinese speakers in their perception of artificial social agents

Bachelor thesis (2023) - J. Hensman, N. Albers, W.P. Brinkman, O.E. Scharenborg

Artificial social agents (ASAs) are systems designed to interact with humans in a socially intelligent manner. As the field of robotics is rapidly advancing, some studies focused on creating more effective agents by analysing how people perceive them. However, culture affects peo ...

Low complexity crosstalk cancellation algorithm for consumer audio systems

Optimizing crosstalk cancellation from a human sound perception perspective

Master thesis (2023) - R.J. Eveleens, Jorge Martinez, A. Noroozi, O.E. Scharenborg, R.T. Rajan

Over the past decade, spatial audio awareness evolved into an in-demand feature in audio entertainment. The addition of sound source locations to, for instance, movies or music adds a level of auditory envelopment and spatial awareness to the audio experience. Expensive setups pr ...

Attention-based deep learning for DNA repair outcome prediction

Learning how the cell repairs DNA breaks using local sequence context

Master thesis (2022) - J.H.D. de Boer, Joana Gonçalves, M.J.T. Reinders, C.F. Seale, O.E. Scharenborg

Recent advancements in quantification of repair outcomes of CRISPR-Cas9 mediated double-stranded DNA breaks (DSBs) have allowed for the use of machine learning for predicting the frequencies of these repair outcomes. Local DNA sequence context influences the frequencies of mutati ...

Decoding Covert Speech from EEG

Development of a novel database containing EEG and audio signals during Dutch covert and overt speech

Master thesis (2022) - B. Dekker, A.C. Schouten, O.E. Scharenborg, M. Kok, H.S. Hung

To enable communication for patients who have lost the ability to speak due to severe neuromuscular diseases, covert speech based brain-computer interfaces (BCIs) might be used. These system use neural signals arising from covert speech and translate them into text or synthesised ...

To enable communication for patients who have lost the ability to speak due to severe neuromuscular diseases, covert speech based brain-computer interfaces (BCIs) might be used. These system use neural signals arising from covert speech and translate them into text or synthesised speech. Covert speech is imagining to speak without moving any of the articulators and therefore does not rely on actual motor activity. As recognizing covert speech from neural signals is extremely challenging, machine learning algorithms are deployed. To make use of the full potential of machine learning approaches in the field of decoding covert speech and to accommodate real-world deployment of a BCI, a large number of training samples is required to train the networks.
In this study, a novel database is presented containing EEG and audio data from 20 subjects recorded during the covert and overt pronunciation of 15 Dutch prompts. To validate the recorded data, two speaker-independent classification tasks were performed using a ResNet-50 algorithm as classifier with spatial-spectral-temporal features extracted from the EEG signals. The speaker-independent three-class classification of pre-stimulus (rest) trials versus covert speech trials versus overt speech trials obtained an average accuracy of 70.6% and the speaker-independent five-class classification of five covert vowels (“aa”, “ee”, “oo”, “ie”, “oe”) obtained an average accuracy of 19.6%. Even though the five-class classification task did not reach an above chance level accuracy, the high performance reached by the three-class classification task provides support of the existence of discriminative information in the covert speech segments to decode covert speech in the future.
Future research should focus on EMG artifact detection and on determining the performance per subject to improve the dataset. Furthermore, subject normalisation strategies should be investigated to address the challenges of subject-independent covert speech decoding.

Everyday Locations as Cues to Smoke

Personalized Environments in Virtual Reality to Elicit Smoking Cravings

Master thesis (2022) - A. Antoniades, W.P. Brinkman, N. Albers, O.E. Scharenborg

Smoking is a leading risk factor negatively impacting the health of people, not only those partaking in it first-hand, but also to those around them. Different methods are available to assist people with quitting smoking, with various degrees of effectiveness. Researchers develop ...

For your voice only

Exploiting side channels in voice messaging for environment detection

Master thesis (2021) - A. Ravindranath, M. Conti, Matteo Cardaioli, O.E. Scharenborg, B. Taebi

Voice messages are an increasingly well-known method of communication, accounting for more than 200 million messages a day. Sending audio messages requires a user to invest lesser effort compared to texting while enhancing the meaning of the message by adding an emotional context ...

Security Evaluation of GoQuorum-based Smart Contracts

A Case Study of Malfunctioning Access Control and Double-Spending

Bachelor thesis (2021) - B.C. Slager, K. Liang, O.E. Scharenborg

GoQuorum is an enterprise blockchain platform that supports smart contracts and allows for private transactions. Smart contracts enable automated payment while eliminating the need for third-party involvement. Previous attacks on smart contracts have already shown that existing v ...

Phon Times

Improving Dutch phoneme recognition

Master thesis (2021) - R.W. Levenbach, O.E. Scharenborg

In this research, Dutch phoneme recognition (PR) is researched and improved. The last research on Dutch PR dates back to 1995. This research presents Dutch PR in modern daylight by researching state-of-the-art techniques found in research on other languages and implementing them ...

In this research, Dutch phoneme recognition (PR) is researched and improved. The last research on Dutch PR dates back to 1995. This research presents Dutch PR in modern daylight by researching state-of-the-art techniques found in research on other languages and implementing them on Dutch PR. The goal of this research is to find the current best architectures for Dutch phoneme recognition, analyse how those architectures perform on different speaking styles, and analyse the errors of the phonemes.
To achieve these goals, four state-of-the-art Neural Network architectures (BLSTM, OPGRU, TDNN-BLSTM, and TDNN-OPGRU) are built and trained on a Dutch dataset containing 15 different speaking styles. Then, the trained architectures are used for three experiments. The first experiment tests the architectures on a 15-speaking style containing the same 15 speaking styles as the training set. The results of that training set are used to calculate the phoneme error rate (PER), analyse the errors, and identify error-prone phonemes. The second and third experiments follow the same methodology on the same-trained architectures, however, are tested on a clear speaking style test set and a spontaneous speaking style test set. The results are used to analyse and compare the performances of the models on different speaking styles. This research shows that the BLSTM and OPGRU are a poor choice for Dutch PR, as the results show relatively high PERs for these two architectures. The hybrid acoustic models TDNN-BLSTM and TDNN-OPGRU show to be good models for Dutch PR, as they gain low PERs. The results in this research set a new benchmark for Dutch PR, as the TDNN-BLSTM and TDNN-OPGRU show the lowest recognition accuracy found in existing Dutch PR literacy. Additionally, this research shows that recognising Dutch clear speech is less complex than recognising Dutch spontaneous speech, which is in line with existing word- and phoneme recognition research on other languages. Besides that, this research makes recommendations for which of the hybrid models is more effective to use when using the different speaking styles as input. Finally, this research finds the phonemes /I/, /a/, /k/, /s/, and the schwa as error-prone phonemes in Dutch PR, as they have high, above average individual PERs and contribute above-average to the overall PER on both the TDNN-BLSTM and TDNN-OPGRU. This research also makes recommendations on where future Dutch PR could focus on. Setting a benchmark for Dutch PR, analysing different speaking styles, and identifying error-prone phonemes hopefully triggers future researchers on further improving Dutch PR.

Machine-learning pipelines for classification of pathological tremor patients

A proof-of-concept

Master thesis (2021) - A. Assis de Souza, W. Mugge, M. Kok, F.C.T. van der Helm, O.E. Scharenborg

Parkinson’s Disease (PD), Essential tremor (ET), and dystonia are movement disorders often misdiagnosed as one another and commonly present tremor as one of their motor symptoms. Rates of misdiagnosis between 30 and 50% of ET patients have been reported, where dystonia and PD are ...

Secure Proximity Detection and Verification

Addressing vulnerabilities in IEEE 802.15.4z UWB

Master thesis (2021) - V. Subramanian, M. Conti, O.E. Scharenborg, A. Zarras

We live in a world where much of our interactions with the environment around us depend on us being physically close to them. For instance, we have proximitybased tokens (e.g., keys and smartcards) for access systems installed at various places such as in cars, at contactless pa ...

We live in a world where much of our interactions with the environment around us depend on us being physically close to them. For instance, we have proximitybased tokens (e.g., keys and smartcards) for access systems installed at various places such as in cars, at contactless payment terminals, and in electronic passports. Moreover, such systems exist in critical environments like nuclear power plants.Unfortunately, the current systems used to detect proximity between devices and/or users are rife with vulnerabilities. Numerous attacks, such as Relay attack, Preamble Injection attack, Early Detect/Late Commit, and Cicada, exist that let an attacker maliciously alter the measured distance. The research community has proposed several solutions to address these problems and based on their inputs, the IEEE 802.15.4a standard was recently amended. Nevertheless, we show that the newer amendment(i.e., IEEE 802.15.4z) is however not entirely secure and still vulnerable to being exploited.In this work, we evaluate and address the vulnerabilities present in the recently introduced standard,IEEE 802.15.4z amendment for UltraWide Band (UWB). This standard forms the basis of proximity detection in a majority of new devices such as keyfobs for cars, access control systems, smartphones like Samsung S21 and Google Pixel 6, and even medical equipment to monitor patients. First, we mount two attacks, namely the CicadaTF and the Adaptive Injection, against UWB based proximity detection systems. Second, we propose a novel approach to detect the presence of these attacks. We create a real world testbed using DWM3000 ICs mounted on NRF52840devkits to launch the attacks and implement our proposed detection approach. We evaluate the efficacy of our approach in three different environments: an indoor residence, a large outdoor passageway, and an office space. These environments were selected to represent the most commonly used places and were based on the802.15.4a channel models document by IEEE. Our experiment results show that the proposed model can detect the presence of attacks with high accuracy (94%) in all three environments. To the best of our knowledge, this is the first research work that presents a way to detect the presence of such attacks and also to be verified on hardware.

Word recognition in a model of visually grounded speech

An analysis using techniques inspired by human speech processing research

Master thesis (2020) - J.S.M. Scholten, O.E. Scharenborg, Danny Merkx, N. Tintarev, Catharine Oertel

A Visually Grounded Speech model is a neural model which is trained to embed image caption pairs closely together in a common embedding space. As a result, such a model can retrieve semantically related images given a speech caption and vice versa. The purpose of this research is ...

Evaluating Image2Speech

The evaluation of automatically generated phoneme captions for images

Master thesis (2020) - J.R.T.E. van der Hout, O.E. Scharenborg

Image2Speech is the relatively new task of generating a spoken description of an image. Similar to Automatic Image Captioning, it is a task focused on describing images, however it avoids the usage of textual resources. An Image2Speech system produces a sequences of phonemes inst ...

Image2Speech is the relatively new task of generating a spoken description of an image. Similar to Automatic Image Captioning, it is a task focused on describing images, however it avoids the usage of textual resources. An Image2Speech system produces a sequences of phonemes instead of (written) words which makes the Image2Speech task applicable to languages which do not have a standardized writing system. This thesis presents an investigation into the evaluation of the Image2Speech task. The Image2Speech output is evaluated with human evaluators as well as multiple objective evaluation metrics. These metrics are often used in the field of Natural Language Processing, such as BLEU, METEOR, PER, etc. and can be used to give an indication of the semantic similarity between two sentences of words. Since humans are the end users of Image2Speech systems, the objective evaluation metrics are correlated with human evaluation in order to determine which metric can best evaluate an Image2Speech system with the end users in mind. For this, first an Image2Speech system was implemented which generates image captions consisting of phoneme sequences. This system outperformed the original Image2Speech system on the Flickr8k corpus, which is a dataset containing 8,000 images which each image also having five written and spoken captions. Subsequently, these phoneme captions were converted into sentences of words in order to be more easily interpretable for human evaluators. The captions were rated by human evaluators for their goodness of describing the image and correlated with the objective evaluation metrics. Although BLEU4 does not perfectly correlate with human ratings, it obtained the highest correlation among the investigated metrics, and is the best currently existing metric for automatically evaluating the Image2Speech task. Current metrics are limited by the fact that they assume their input to be words. A more appropriate metric for the Image2Speech task should assume its input to be parts of words, e.g. phonemes, instead.

Graph-Time Convolutional Neural Network

Learning from Time-Varying Signals defined on Graphs

Master thesis (2020) - G. Mazzola, E. Isufi, O.E. Scharenborg, G.J.T. Leus, Femke C. Vossepoel

Time-varying network data are essential in several real-world applications, such as temperature forecasting and earthquake classification. Spatial and temporal dependencies characterize these data and, therefore, conventional machine learning tools often fail to learn these joint ...

Time-varying network data are essential in several real-world applications, such as temperature forecasting and earthquake classification. Spatial and temporal dependencies characterize these data and, therefore, conventional machine learning tools often fail to learn these joint correlations from data. On the one hand, hybrid models to learn from time-varying network data combine several specialized models able to capture these dependencies separately, thus ignoring their joint spatio-temporal interactions. On the other hand, state-of-the-art approaches for jointly learning time-varying network data do not exploit the useful prior provided by a graph-time product graph. This prior structural knowledge can aid learning and help the models improve their performance. For this reason, we propose a novel neural network architecture to learn from time-varying network data using product graphs and graph convolutions, thus exploiting this prior during learning. In particular, our architecture (i) learns the most suitable graph-time product graph to represent the time-varying network data and model the graph-time interactions; (ii) performs graph convolutions over this product graph to learn the graph-time interactions from data; (iii) employs graph-time pooling to reduce the dimensionality over layers. To the best of our knowledge, no research has yet attempted to capture spatial and temporal dependencies using graph convolutions over product graphs. Results on synthetic and real-world data show that the proposed method is useful in learning from time-varying network data for both regression and classification tasks. For classification, we cure a real-world dataset for earthquake classification and compare the proposed approach with state-of-the-art models. Results indicate that graph-aware models outperform graph-unaware models on this task. For regression, we evaluate the proposed method on two real-world temperature datasets and compare our architecture with graph-aware and graph-unaware models. Results on the smaller dataset show that linear models outperform neural-network models. On the larger dataset, we find that prior knowledge about graph-time interactions seems to be less beneficial in case of abundance of data. For the synthetic experiments, we find that (i) learning the structure of the graph-time product graph from data improves the performance compared to adopting a fixed type of product graph; (ii) learning sparser graph-time product graphs further improves the performance; (iii) the proposed graph-time pooling technique contributes to the model's generalization capabilities.

Project Delphi

What is Innovation?

Bachelor thesis (2019) - L. Boleij, T.J.B. Hartman, M.R.F. Jorden, W. Chen, O.E. Scharenborg

Being a company with over 1500 employees, a lot of data is available about people and their day-to-day pursuits, including projects they are working on. Company X has requested to gain more insight into this data, as it is currently scattered over multiple systems. Specifically, ...

Retrieval-Based Open-Domain Question Answering

Exploring The Impact on The Retrieval Component across Datasets

Master thesis (2019) - K. Jiang, C. Hauff, O.E. Scharenborg, F.A. Kuipers

Open-domain question answering (QA) is an important step in Artificial Intelligence and its ultimate goal is to build a QA system that can answer any question posed by humans. The majority of the open-domain QA system is the retrieval-based open-domain QA system, which enables th ...

Open-domain question answering (QA) is an important step in Artificial Intelligence and its ultimate goal is to build a QA system that can answer any question posed by humans. The majority of the open-domain QA system is the retrieval-based open-domain QA system, which enables the retrieval component to retrieve relevant documents from a large-scale knowledge source to a question and the answer extraction component to extract the answer to this question based on retrieved documents. As the techniques of Deep Learning progressing significantly, many researchers tried to apply the neural reading comprehension (RC) model to serve the answer extraction component of the open-domain QA system. However, the performance of the neural RC model in open-domain QA is considerably worse than the performance of it in RC-style QA. Therefore, many works have focused on the neural RC model for addressing the performance gap, whereas the retrieval component of the open-domain QA system lacks equivalent attention. Some researchers have built the neural network based information retrieval (IR) models, but currently, it is still difficult for these neural IR models to directly retrieve documents from a large-scale knowledge source in open-domain QA. Hence, many works attempted to use neural IR models for re-ranking documents retrieved by the traditional but efficient IR models (e.g., TF-IDF, BM25) in open-domain QA. However, these works did not analyze the impact of different questions of QA datasets on traditional IR models. Thus, this research gap is the focus in this thesis. We conduct error analyses of questions of different QA datasets to figure out the error types of questions that have a negative impact on the traditional IR models. From the error analysis, we learn that different QA datasets have different impacts on the traditional IR models and are differently hard to be dealt with by the traditional IR models. Therefore, we propose hypotheses that might mitigate the negative impact of the error types of questions that are relatively harder to be handled by the traditional IR models. Furthermore, we perform experiments based on the methodologies that implement our hypotheses for figuring out the validity of these hypotheses. In conclusion, we believe that our work is a step forward to obtaining more insights into the retrieval component of the open-domain QA system and will contribute to the development of the retrieval component for a better open-domain QA system. Moreover, our work can give our users guidance on how to issue a more suitable question that can be processed by the open-domain QA system for giving a more accurate and better answer.

Rotation invariant filters in CNNs

Applied to segmentation of aerial images for land-use classification

Master thesis (2018) - A. Dhar, J.C. van Gemert, M.J.T. Reinders, O.E. Scharenborg, Daan van der Maas

Convolutional neural networks are showing incredible performance in image classification, segmentation, object detection and other computer vision applications in recent years. But they lack understanding of affine transformations to input data. In this work, we introduce rotatio ...

O.E. Scharenborg

Authored

AnyoneNet

Synchronized Speech and Talking Head Generation for Arbitrary Persons

Audio-Visual Speech Recognition in MISP2021 Challenge

Dataset Release and Deep Analysis

Why listening in background noise is harder in a non-native language than in a native language

A review

Contributed