Gesture in the eye of the beholder

Gesture in the eye of the beholder: An eye-tracking study on factors determining the attention for gestures produced by people with aphasia

van Nispen, K. (TU Delft Instituut voor Talen en Academische Vaardigheden; Tilburg University)
Sekine, Kazuki (Waseda University)
van der Meulen, Ineke (Erasmus MC; Rijndam Rehabilitation Center)
Preisig, Basil C. (University of Zürich)

Instituut voor Talen en Academische Vaardigheden

2022

Co-speech hand gestures are an ubiquitous form of nonverbal communication, which can express additional information that is not present in speech. Hand gestures may become more relevant when verbal production is impaired, as in speakers with post-stroke aphasia. In fact, speakers with aphasia produce more gestures than non-brain damaged speakers. Furthermore, there is evidence to suggest that speakers with aphasia produce gestures that convey information essential to understand their communication. In the present study, we addressed the question whether these gestures catch the attention of their addressees. Healthy volunteers (observers) watched short video clips while their eye movements were recorded. These video clips featured speakers with aphasia and non-brain damaged speakers describing two different scenarios (buying a sweater or having witnessed an accident). Our results show that hand gestures produced by speakers with aphasia are on average attended to longer than gestures produced by non-brain damaged speakers. This effect was significant even when we controlled for the longer duration of the gestural movements in speakers with aphasia. Further, the amount of information in speech was also correlated with gesture attention. That is gestures produced by speakers with less informative speech were attended to more frequently. In conclusion, our findings suggest that listeners reallocate their attention and focus more strongly on non-verbal information from co-speech gestures if speech comprehension becomes challenging due to the speaker's verbal production deficits. These findings support a communicative function of co-speech gestures and advocate for instructing people with aphasia to communicate things in the form of gestures that cannot be expressed verbally because interlocutors take notice of these gestures.

Aphasia
Attention
Eye movements
Gestures
Multimodal communication
Speech perception

http://resolver.tudelft.nl/uuid:4a2d9ef5-60f7-44d8-b29a-ccf67a8bc842

https://doi.org/10.1016/j.neuropsychologia.2022.108315

0028-3932

Neuropsychologia, 174

Institutional Repository

journal article

© 2022 K. van Nispen, Kazuki Sekine, Ineke van der Meulen, Basil C. Preisig