Health and Medicine
PUBLISHED
Brain-computer interfaces (BCIs) carry a lot of hope for people with certain disabilities, potentially reading the brain activity of people unable to speak and turning it into words. While this technology gets the headlines, BCIs that can convey facial expressions and hand gestures have also been produced, but it is only now that one has been demonstrated to do both at the same time.
“Conversation is about much more than the words being spoken. It’s a multilayered, dynamic process involving the whole motor cortex,” said Professor Edward Chang, M.D. of the University of California, San Francisco in a statement. “This proof-of-concept shows us it’s possible for a BCI to restore some of this freedom and flexibility.”
Chang and colleagues trained decoders to identify the brain activity associated with three amyotrophic lateral sclerosis (ALS) patients’ speech and gestures and had online avatars speak and move as the patients were trying to. Each patient used a vocabulary of 10 words and 10 gestures, giving 100 ways they could be combined.
ALS, sometimes known as motor neuron disease, often triggers brainstem strokes, which cause severe paralysis. These strokes interfere with upper body movement that provides most of our non-verbal communication, as well as the affecting the ability to speak.
Eye-tracking technologies, such as the one famously used by Stephen Hawking are a slow and exhausting way to give people with ALS a little of their voice back, with tone – let alone non-verbal communication – ignored entirely.
Despite fears about where putting chips into people’s brains might eventually lead, those struggles have created enthusiasm for sensors that can read the activity that, were it not for the ALS, would move a patient’s vocal cords and mouth.
Chang and colleagues are one of the teams working on this, previously placing a strip of sensors known as an electrocorticography (ECoG) array onto patients’ motor cortices, and developing computer decoders to translate these into commands. These commands have been used to control a virtual head, which serves as an avatar for the patient.
Some sensors were only triggered by speech, and some by gestures, but the team report that “speech and gesture responses were not fully segregated. A subset of electrodes, particularly in precentral gyrus, were modulated during both behaviors.”
Notably, Chang and co-authors found that attempting to translate the signals associated with speech and gestures independently and then combining them wasn’t very reliable. That’s in keeping with previous studies that have shown the brain signals associated with using both hands at once are different from those you’d get from simply adding the drivers of each hand independently.
Things worked better when the decoders were trained on simultaneous words and accompanying movement, although using a mix of independent and simultaneous communication worked best of all.
“These promising results give me hope that in the future, patients with severe paralysis will be able to recapture the holistic nature of human communication,” said Dr Debara Tucci of the NIH.
The system is still a long way from what science fiction has led us to imagine. Instead of an implantable version communicating by Wi-Fi to an external decoder, the subjects of the study had wires carrying the sensors’ output to attached equipment, creating an additional obstacle to freedom of movement.
Moreover, besides the small vocabularies, even the best-trained decoders the team experimented with still mistranslated a sixth to a third of the word/gesture combinations.
The study is published in Nature Neuroscience.
