Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments.

Full Abstract

Viewing a speaker's articulatory movements substantially improves a listener's ability to understand spoken words, especially under noisy environmental conditions. It has been claimed that this gain is most pronounced when auditory input is weakest, an effect that has been related to a well-known principle of multisensory integration--"inverse effectiveness." In keeping with the predictions of this principle, the present study showed substantial gain in multisensory speech enhancement at even the lowest signal-to-noise ratios (SNRs) used (-24 dB), but it was also evident that there was a "special zone" at a more intermediate SNR of -12 dB where multisensory integration was additionally enhanced beyond the predictions of this principle. As such, we show that inverse effectiveness does not strictly apply to the multisensory enhancements seen during audiovisual speech perception. Rather, the gain from viewing visual articulations is maximal at intermediate SNRs, well above the lowest auditory SNR where the recognition of whole words is significantly different from zero. We contend that the multisensory speech system is maximally tuned for SNRs between extremes, where the system relies on either the visual (speech-reading) or the auditory modality alone, forming a window of maximal integration at intermediate SNR levels. At these intermediate levels, the extent of multisensory enhancement of speech recognition is considerable, amounting to more than a 3-fold performance improvement relative to an auditory-alone condition.

Learn Faster Today Improve your study skills

Author information

Author/s: Ross, Lars A (LA); Saint-Amour, Dave (D); Leavitt, Victoria M (VM); Javitt, Daniel C (DC); Foxe, John J (JJ);

Affiliation: Program in Cognitive Neuroscience, Department of Psychology, The City College of the City University of New York, New York, NY 10031, USA.

Grants: AG22696 (Agency:NIA NIH HHS) ; MH65350 (Agency:NIMH NIH HHS) ; R01 MH065350-05 (Agency:NIMH NIH HHS)

Journal and publication information

Publication Type: Journal Article; Research Support, N.I.H., Extramural

Journal: Cerebral cortex (New York, N.Y. : 1991) (Cereb Cortex), published in United States. (Language: eng)

Reference: 2007-May; vol 17 (issue 5) : pp 1147-53

Dates: Created 2007/04/05; Completed 2007/05/31; Revised 2007/11/14;

PMID: 16785256, status: MEDLINE (last retrieval date: 12/26/2008)

Sourced from the National Library of Medicine. Abstract text and other information may be subject to copyright.

External Links for this article (including full text providers, if available):

Click Electronic Full-text Provider Links to see options for finding the electronic full text links to this article. Note there may be a subscription or fee required for access to the full text. See our FAQ for information on finding FREE full text articles.

This article may also be located in paper journal collections available in many libraries. Use the Journal and Publication Information above to find the full article.

MeSH headings (categories)

This article was linked to the MESH Headings shown below.

Adolescent
Adult
Artifacts
Auditory Perception - physiology
Cognition - physiology
Facial Expression
Female

Female
Humans
Male
Middle Aged
Speech Perception - physiology
Task Performance and Analysis
Visual Perception - physiology

These are the highest related articles currently in the database:

See 100+ related articles.