2024P The timing of an avatar’s gestures differentially influences lexical stress perception in normal and simulated cochlear implant hearing conditions

Cochlear implants (CI) alter the transmission of sound, degrading fundamental frequency information, disrupting the perception of lexical stress. Providing additional (even artificially generated) visual cues could perhaps support CI users in speech perception. Recent studies showed that lexical stress perception is influenced by beat gestures: the same acoustic token (e.g., “content”) is more likely to be perceived with a strong-weak (SW) stress pattern (e.g., the noun “CONtent”) when the beat falls on its first syllable, and with a weak-strong (WS) stress pattern (e.g., the adjective “conTENT”) when the beat falls on its second syllable (“manual McGurk effect”). The present study investigated whether beat gestures made by an avatar are beneficial when listening to vocoded speech, simulating CI conditions. Normal-hearing participants watched videos of a gesturing avatar, while a member of a disyllabic Dutch minimal pair was produced with a clear SW, a clear WS, or an ambiguous stress pattern, in normal or 8-channel tone-vocoded speech. The avatar’s beat gesture fell on the first or second syllable of the word. With normal speech, beat gestures biased lexical stress perception mostly in the clear SW and ambiguous stress patterns. With vocoded speech, the manual McGurk effect was observed in the clear WS and ambiguous stress patterns. Since WS is an infrequent stress pattern in Dutch, the avatar’s beat gestures appear to modulate lexical stress perception in conjunction with internal models of frequency of occurrence and the clarity of the speech signal. The present findings could inspire the development of avatars gesturing to the prosody of an interlocutor to support speech recognition in CI users.