Decoding Covert Speech from EEG

Development of a novel database containing EEG and audio signals during Dutch covert and overt speech

Master thesis (2022)

Authors

B. Dekker Mechanical Engineering

Contributors

A.C. Schouten Biomechatronics & Human-Machine Control - Mechanical, Maritime and Materials Engineering (supervisor 1)
O.E. Scharenborg (supervisor 1)
M. Kok Team Manon Kok - Mechanical, Maritime and Materials Engineering (supervisor 2)
H.S. Hung Pattern Recognition and Bioinformatics - EEMCS (supervisor 2)
Faculty
Mechanical Engineering
Copyright: © 2022 Bo Dekker
ResNet Covert speech Electroencephalography (EEG) Brain-computer interface
More Info
expand_more

Published Date

25-10-2022

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Mechanical Engineering

Abstract

To enable communication for patients who have lost the ability to speak due to severe neuromuscular diseases, covert speech based brain-computer interfaces (BCIs) might be used. These system use neural signals arising from covert speech and translate them into text or synthesised speech. Covert speech is imagining to speak without moving any of the articulators and therefore does not rely on actual motor activity. As recognizing covert speech from neural signals is extremely challenging, machine learning algorithms are deployed. To make use of the full potential of machine learning approaches in the field of decoding covert speech and to accommodate real-world deployment of a BCI, a large number of training samples is required to train the networks.
In this study, a novel database is presented containing EEG and audio data from 20 subjects recorded during the covert and overt pronunciation of 15 Dutch prompts. To validate the recorded data, two speaker-independent classification tasks were performed using a ResNet-50 algorithm as classifier with spatial-spectral-temporal features extracted from the EEG signals. The speaker-independent three-class classification of pre-stimulus (rest) trials versus covert speech trials versus overt speech trials obtained an average accuracy of 70.6% and the speaker-independent five-class classification of five covert vowels (“aa”, “ee”, “oo”, “ie”, “oe”) obtained an average accuracy of 19.6%. Even though the five-class classification task did not reach an above chance level accuracy, the high performance reached by the three-class classification task provides support of the existence of discriminative information in the covert speech segments to decode covert speech in the future.
Future research should focus on EMG artifact detection and on determining the performance per subject to improve the dataset. Furthermore, subject normalisation strategies should be investigated to address the challenges of subject-independent covert speech decoding.