Scientists have developed a machine that can monitor a patient’s brain activity, and reconstruct sentences that they have been listening to. According to the researchers, this is the first time any study has shown the real-time decoding of sentences based solely on neural activity.
When someone is listening to another person speak, a region of the brain known as the superior temporal gyrus (STG) fires up with neural activity. This activity can be used to decode a whole host of information, including the acoustic properties of the speech. But it can also, amazingly, be used to reconstruct the phoneme sequences, and according to this latest study, the pattern of vowels and consonants.
In a paper published in the Journal of Neural Engineering, researchers describe how they have been able to monitor the brain activity of someone, and then use it to reconstruct the sentence that they have just been listening to in real time.
In a recent similar study, one that has not been peer-reviewed yet, a team taught a machine to reconstruct what a person sees, by reading their brain activity. The team used a deep neural network (DNN) to decode brain signals recorded by a fMRI scanner in order to produce a computer-generated reconstructed image of what the participants saw.
This study has so far been conducted on two patients, and currently requires them to have neural implants placed directly onto their brain surface. They then listened to the same 10 sentences being read to them repeatedly, while the electrodes monitored the activity in the STG. The computer software, called real-time Neural Speech Recognition (rtNSR) was then able to decode the activity in – you guessed it – real time and generate the sentences.
The different sentences included a mixture of statements (“Nobody likes snakes”), questions (“Junior, what on Earth’s the matter with you?”) and even quotes (“A bullet,” she answered.”). According to the study, the rtNSR was at least 90 percent accurate during each test regardless of what was being said, needing just seven minutes of training data on average to then decode the words.
The researchers hope that this kind of technology, which is still in fairly early stages, could be used in a medical setting, particularly where patients are either paralyzed or otherwise unable to speak. If it can be further developed it could prove revolutionary.
There are concerns, however, that such a machine could move us dangerously close to a Black Mirror-esque world in which the device might accidentally transcribe someone's secret thoughts. Currently, however, it is limited to figuring out what a person is thinking when they’re read the same sentence repeatedly, so thankfully that kind of dystopian future is hopefully still some way off.
1