TECHNOLOGY
Back in 2008, theoretical physicist Stephen Hawking used a speech synthesiser program on an old Apple II computer to "talk”. He relied on hand controls to work the system, which became more difficult as his Lou Gehrig's disease progressed. When he upgraded to a new device, called a cheek switch, it detected when Hawking tensed the muscle in his cheek, helping him speak, write emails, or surf the web. At the time, it was state of the art.
Now, twelve years later, neuroscientists at the University of California, San Francisco, have come up with an artificial intelligence (AI)-based system that can decode brain waves into written text faster than existing technologies. In other words, it can turn thoughts into text.
“We are not there yet, but we think this could be the basis of a speech prosthesis,” said Dr Joseph Makin, who co-wrote the report, which appeared in Nature Neuroscience in late March.
Makin and his colleagues recruited four participants and implanted electrode arrays in their brains to monitor epileptic seizures. Each of the four read 50 sentences aloud many times, including lines such as “Tina Turner is a pop singer” and “There is chaos in the kitchen”. As each spoke, their brain activity was monitored and the data collected and fed into a machine-learning algorithm, a type of AI that converts brain activity data for each spoken sentence into a string of numbers. To make sure the numbers related only to aspects of speech, the system compared sounds predicted from small chunks of brain activity data with actual recorded audio. The string of numbers was then fed into a second part of the system, which converted it into a sequence of words.
At the outset, the system spat out nonsense sentences, among them: “Those musicians harmonise marvellously” was decoded as “The spinach was a famous singer”, and “A roll of wire lay near the wall” became “Will robin wear a yellow lily”. Some translated with improper grammar, such as "several adults the kids was eaten by” and quite philosophical-sounding sentences, such as "the oasis was a mirage”. But as the system compared each sequence of words with the sentences that were actually read aloud, it improved, learning how the string of numbers related to words, and which words tend to follow each other. Over time, the accuracy of the system has improved notably. For instance, in one case, the system got 97 percent of the sentences correct, a better result than the average human transcriber.
Dr Christian Herff, an assistant professor in the department of neurosurgery at Maastricht University in The Netherlands, who was not involved in the study, said the research was exciting because the system used less than 40 minutes of training data for each participant, and a limited collection of sentences, rather than the millions of hours typically needed. “By doing so, they [are achieving] levels of accuracy that haven’t been achieved so far,” he said.
However, he noted the system won’t work for many severely disabled patients or those with speech disorders as it relied on brain activity recorded from people speaking a sentence out loud. “This is not translation of thought [but of brain activity involved in speech].” Herff added that people shouldn’t worry about others reading their thoughts just yet: brain electrodes would have to be implanted, while imagined speech is very different to inner voice.
And while the algorithm is only processing a small number of sentences and words compared to what a user would ultimately want, as more gains are made, AI could one day help millions of people with speech disabilities communicate with far greater ease.
There’s just a word of warning, though. Dr Mahnaz Arvaneh, an expert in brain machine interfaces at Sheffield University, argues it’s important to consider ethical issues now. “We [are still] very, very far away from the point where machines can read our minds,” she said. “But it doesn’t mean we should not think about it and plan for it.”