2024 Lexical stress influences the perceived timing of beat gestures

Language is multimodal: for example, speech is often accompanied by hand gestures. Although beat gestures do not convey meaning themselves, they frequently co-occur with prosodic prominence. Thus, they can indicate stress in a word and hence influence spoken-word recognition. However, little is known about the reverse relationship. The current study investigated whether lexical stress has an effect on the perceived timing of hand beats. We used videos in which a disyllabic word, embedded in a carrier sentence (Exp 1) or in isolation (Exp 2), was coupled with an up-and-down hand beat, while varying their degrees of asynchrony. Using a novel beat timing estimation task, Experiment 1 revealed that gestures were estimated to occur closer in time to the pitch peak in a stressed syllable than their actual timing, hence a reduced temporal difference between gestures and stress. Experiment 2, a canonical 2AFC task, further demonstrated that listeners tended to perceive a gesture, falling midway between two syllables, on the syllable receiving stronger cues to stress than the other. The temporal attraction effect of stress on perceived gestural timing was greater when gestural timing was most ambiguous, and was driven by f0 and intensity. This study provides new evidence for auditory influences on visual perception, supporting bidirectionality in audiovisual interaction between speech-related signals that occur in everyday face-to-face communication.