Decoding Covert Speech from EEG

Decoding Covert Speech from EEG: Development of a novel database containing EEG and audio signals during Dutch covert and overt speech

Dekker, Bo (TU Delft Mechanical, Maritime and Materials Engineering)

Schouten, A.C. (mentor)
Scharenborg, O.E. (mentor)
Kok, M. (graduation committee)
Hung, H.S. (graduation committee)

Delft University of Technology

Biomedical Engineering

2022-10-25

To enable communication for patients who have lost the ability to speak due to severe neuromuscular diseases, covert speech based brain-computer interfaces (BCIs) might be used. These system use neural signals arising from covert speech and translate them into text or synthesised speech. Covert speech is imagining to speak without moving any of the articulators and therefore does not rely on actual motor activity. As recognizing covert speech from neural signals is extremely challenging, machine learning algorithms are deployed. To make use of the full potential of machine learning approaches in the field of decoding covert speech and to accommodate real-world deployment of a BCI, a large number of training samples is required to train the networks.
In this study, a novel database is presented containing EEG and audio data from 20 subjects recorded during the covert and overt pronunciation of 15 Dutch prompts. To validate the recorded data, two speaker-independent classification tasks were performed using a ResNet-50 algorithm as classifier with spatial-spectral-temporal features extracted from the EEG signals. The speaker-independent three-class classification of pre-stimulus (rest) trials versus covert speech trials versus overt speech trials obtained an average accuracy of 70.6% and the speaker-independent five-class classification of five covert vowels (“aa”, “ee”, “oo”, “ie”, “oe”) obtained an average accuracy of 19.6%. Even though the five-class classification task did not reach an above chance level accuracy, the high performance reached by the three-class classification task provides support of the existence of discriminative information in the covert speech segments to decode covert speech in the future.
Future research should focus on EMG artifact detection and on determining the performance per subject to improve the dataset. Furthermore, subject normalisation strategies should be investigated to address the challenges of subject-independent covert speech decoding.

Covert speech
ResNet
Electroencephalography (EEG)
Brain-computer interface

http://resolver.tudelft.nl/uuid:96f43f80-41da-4e14-ba83-6cca31309fbb

Student theses

master thesis

© 2022 Bo Dekker