Search Here

When it comes to learning and understanding language, researchers have known that the ability to hear is extremely important. If one cannot hear words, one cannot repeat them correctly. Even though this association between speech and sound is strong, researchers have also found that the role of vision is important as well. In a new study, researchers looked into how the brain processes language through the use of sound and sight. They discovered that vision plays a bigger role than previously believed.

For this study, researchers wanted to examine the phenomenon called the McGurk effect in more depth. The McGurk effect, which was named after Scottish cognitive psychologist Harry McGurk, who conducted several studies in the 1970s, states that when the sound does not match the sight, the brain chooses to rely on vision in order to understand the situation. The research team of bioengineers from the University of Utah created three scenarios that allowed them to monitor brain activity and signals in the temporal cortex, which is a part of the brain tied to processing sounds.

The researchers recruited four volunteers, two men and two women, who suffered from severe epilepsy. The participants were already undergoing surgery to treat their epileptic seizures. The researchers placed electrodes on the left, right or both sides of the brain hemispheres on each individual. The participants then watched three scenarios focused on a mouth. In the first scenario, the mouth spoke the sounds "ba," "va," "tha" and "ga." The sound and the sight matched. In the second video, the sounds and the mouth movement did not match at all. In the last video, the sound and sight also did not match but the difference was barely noticeable. For example, the mouth might have said "va" but the sound played "ba."

"For the first time, we were able to link the auditory signal in the brain to what a person said they heard when what they actually heard was something different. We found vision is influencing the hearing part of the brain to change your perception of reality-and you can't turn off the illusion," said the study's first author, Elliot Smith, a bioengineering and neuroscience graduate student at the University of Utah. "People think there is this tight coupling between physical phenomena in the world around us and what we experience subjectively, and that is not the case."

The researchers found that in scenarios one and two, the participants' brain activity indicated that they relied on their sound processing to understand what was spoken. In the third scenario, however, the researchers found that the participants relied on their vision to understand what was spoken. The researchers believe that their findings indicate a stronger relationship between understanding language and vision. They hope that their study could help with future treatment developments for people with impaired language processing.

Smith worked with bioengineer Bradley Greger from the University of Utah and neurosurgeon Paul House. The study was published in PLOS ONE.