2023 Handy prosody: how hands can help you hear

Speech conveys both segmental information about vowels and consonants as well as suprasegmental information about for instance intonation, rhythm, and lexical stress; also known as the prosody of speech. However, in face-to-face conversations, we do not only exchange sounds; we also move, nod, and gesture to the rhythm of our speech. In this keynote, I will demonstrate how the timing of hand gestures contributes to audiovisual prosody perception, with a focus on lexical stress. For instance, evidence for a ‘manual McGurk effect’ showcases how even relatively simple flicks of the hands can guide whether you hear "PLAto" or "plaTEAU". Moreover, human listeners are shown to actively weigh various multisensory cues to prosody depending on the listening conditions at hand. Thus, these findings emphasize that prosody is a multimodal linguistic phenomenon, with the voice, lips, and even hands conveying prosody in concert.